Method and apparatus for an adaptive and interactive teaching of playing a musical instrument

ABSTRACT

Embodiments pertain to a method for use with a musical piece, or part thereof, comprising: presenting, by at least one client device to a plurality of users, a corresponding plurality of sequences of musical symbols to be cooperatively played by the plurality of users using a plurality of instruments; capturing instrument outputs produced by the plurality of instruments to generate audio data descriptive of the captured instrument outputs; analyzing the audio data for determining a level of correspondence between the captured instrument outputs and the plurality of musical symbols presented to the plurality of users; and presenting, based on the level of correspondence, an adapted or non-adapted sequence of music symbols to at least one of the plurality of players.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part of U.S. non-provisionalpatent application Ser. No. 17/388,050, filed 29 Jul. 2021, which isrelated to and claims priority from U.S. provisional patent applicationNo. 63/120,434, filed 2 Dec. 2020, and U.S. provisional patentapplication No. 63/162,823, filed 18 Mar. 2021, all of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

This disclosure relates generally to an apparatus and method forteaching of playing of a musical instrument.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are for the purpose of generally presenting the backgroundcontext of the disclosure and are not prior-art to the claims in thisapplication, and are not expressly or impliedly admitted to be prior artby inclusion in this section.

FIG. 1 shows a block diagram that illustrates a system 10 including acomputer system 11, and an associated Internet 22 connection. Suchconfiguration is typically used for computers (hosts) connected to theInternet 22 and executing a server, or a client (or a combination)software. The computer system 11 may be used as a portable electronicdevice such as a notebook/laptop computer, a media player (e.g., MP3based or video player), a desktop computer, a laptop computer, acellular phone, a smartphone, a tablet, or a Personal Digital Assistant(PDA), an image processing device (e.g., a digital camera or videorecorder), any other handheld or fixed location computing devices, or acombination of any of these devices. Note that while FIG. 1 illustratesvarious components of the computer system 11, it is not intended torepresent any particular architecture or manner of interconnecting thecomponents.

Network computers, handheld computers, cell phones and other dataprocessing systems that have fewer or more components, may also be used.For example, the computer system 11 of FIG. 1 may be any personalcomputer. The computer system 11 may include a bus 13, an interconnect,or other communication mechanism for communicating information, and aprocessor 12, commonly in the form of an integrated circuit, coupled tothe bus 13 for processing information, and for executing the computerexecutable instructions. The computer system 11 may also include a mainmemory 15 a, such as a Random Access Memory (RAM), or other dynamicstorage device, coupled to the bus 13 for storing information andinstructions to be executed by the processor 12. The main memory 15 aalso may be used for storing temporary variables or other intermediateinformation during execution of instructions to be executed by theprocessor 12.

The computer system 11 further includes a Read Only Memory (ROM) 15 b(or other non-volatile memory) or other static storage device coupled tothe bus 13 for storing static information and instructions for theprocessor 12. A storage device 15 c, may comprise a magnetic disk oroptical disk, such as a hard disk drive (HDD) for reading from andwriting to a hard disk, a Solid State Drive (SSD) for reading from andwriting to a solid state disk, a flash storage for reading and writingfrom flash drive, a magnetic disk drive for reading from and writing toa magnetic disk, an optical disk drive (such as DVD) for reading fromand writing to a removable optical disk, or any combination thereof,that is coupled to the bus 13 for storing information and instructions.The hard disk drive, magnetic disk drive, and optical disk drive may beconnected to the system bus 13 by a hard disk drive interface, amagnetic disk drive interface, and an optical disk drive interface,respectively. The drives and their associated computer-readable mediaprovide non-volatile storage of computer readable instructions, datastructures, program modules and other data for the general-purposecomputing devices.

Typically, the computer system 11 includes an Operating System (OS)stored in a non-volatile storage 15 b for managing the computerresources and provides the applications and programs with access to thecomputer resources and interfaces. An operating system commonlyprocesses system data and user input, and responds by allocating andmanaging tasks and internal system resources, such as controlling andallocating memory, prioritizing system requests, controlling input andoutput devices, facilitating networking and managing files. Non-limitingexamples of operating systems are Microsoft Windows, Mac OS X, andLinux.

The computer system 11 may be coupled via the bus 13 to a display 17,such as a Liquid Crystal Display (LCD), a flat screen monitor, a touchscreen monitor or similar means for displaying text and graphical datato a user. The display 17 may be connected via a video adapter forsupporting the display. The display 17 allows a user to view, enter,and/or edit information that is relevant to the operation of the system10. An input device 18, including alphanumeric and other keys, iscoupled to the bus 13 for communicating information and commandselections to the processor 12. Another type of user input device is acursor control 18 a, such as a mouse, a trackball, or cursor directionkeys for communicating direction information and command selections tothe processor 12 and for controlling cursor movement on the display 17.This cursor control 18 a typically has two degrees of freedom in twoaxes, a first axis (e.g., x) and a second axis (e.g., y), that allowsthe device to specify positions in a plane.

The computer system 11 may be used for implementing the methods andtechniques described herein. According to one embodiment, these methodsand techniques are performed by the computer system 11 in response tothe processor 12 executing one or more sequences of one or moreinstructions contained in the main memory 15 a. Such instructions may beread into the main memory 15 a from another computer-readable medium,such as the storage device 15 c. Execution of the sequences ofinstructions contained in the main memory 15 a causes the processor 12to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the arrangement.Thus, embodiments of the invention are not limited to any specificcombination of hardware circuitry and software.

The term “processor” is used herein to include, but not limited to, anyintegrated circuit or any other electronic device (or collection ofelectronic devices) capable of performing an operation on at least oneinstruction, including, without limitation, a microprocessor (μP), amicrocontroller (μC), a Digital Signal Processor (DSP), or anycombination thereof. A processor, such as the processor 12, may furtherbe a Reduced Instruction Set Core (RISC) processor, a ComplexInstruction Set Computing (CISC) microprocessor, a Microcontroller Unit(MCU), or a CISC-based Central Processing Unit (CPU). The hardware ofthe processor 12 may be integrated onto a single substrate (e.g.,silicon “die”), or distributed among two or more substrates.Furthermore, various functional aspects of the processor 12 may beimplemented solely as a software (or firmware) associated with theprocessor 12.

A memory can store computer programs or any other sequence of computerreadable instructions, or data, such as files, text, numbers, audio andvideo, as well as any other form of information represented as a stringor structure of bits or bytes. The physical means of storing informationmay be electrostatic, ferroelectric, magnetic, acoustic, optical,chemical, electronic, electrical, or mechanical. A memory may be in theform of an Integrated Circuit (IC, a.k.a. chip or microchip).Alternatively or in addition, a memory may be in the form of a packagedfunctional assembly of electronic components (module). Such module maybe based on a Printed Circuit Board (PCB) such as PC Card according toPersonal Computer Memory Card International Association (PCMCIA) PCMCIA2.0 standard, or a Single In-line Memory Module (SIMM) or a Dual In-lineMemory Module (DIMM), standardized under the JEDEC JESD-21C standard.Further, a memory may be in the form of a separately rigidly enclosedbox such as an external Hard-Disk Drive (HDD), an external Solid StateDisk (SSD), or any combination thereof.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor 12 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The bus 13 carries the data to themain memory 15 a, from which the processor 12 retrieves and executes theinstructions. The instructions received by the main memory 15 a mayoptionally be stored on the storage device 15 c either before or afterexecution by the processor 12.

The computer system 11 commonly includes a communication interface 9coupled to the bus 13. The communication interface 9 provides a two-waydata communication coupling to a network link 8 that is connected to aLocal Area Network (LAN) 14. As a non-limiting example, thecommunication interface 9 may be a Local Area Network (LAN) card toprovide a data communication connection to a compatible LAN. Forexample, Ethernet-based connection based on IEEE802.3 standard may beused, such as 10/100BaseT, 1000BaseT (gigabit Ethernet), 10 gigabitEthernet (10GE or 10 GbE or 10 GigE per IEEE Std. 802.3ae-2002asstandard), 40 Gigabit Ethernet (40 GbE), or 100 Gigabit Ethernet (100GbE as per Ethernet standard IEEE P802.3ba). These technologies aredescribed in Cisco Systems, Inc. Publication number 1-587005-001-3(6/99), “Internetworking Technologies Handbook”, Chapter 7: “EthernetTechnologies”, pages 7-1 to 7-38, which is incorporated in its entiretyfor all purposes as if fully set forth herein. In such a case, thecommunication interface 9 typically includes a LAN transceiver or amodem, such as a Standard Microsystems Corporation (SMSC) LAN91C11110/100 Ethernet transceiver, described in the Standard MicrosystemsCorporation (SMSC) data-sheet “LAN91C111 10/100 Non-PCI Ethernet SingleChip MAC+PHY” Data-Sheet, Rev. 15 (02-20-04), which is incorporated inits entirety for all purposes as if fully set forth herein.

An Internet Service Provider (ISP) 16 is an organization that providesservices for accessing, using, or participating in the Internet 22. TheInternet Service Provider 16 may be organized in various forms, such ascommercial, community-owned, non-profit, or otherwise privately owned.Internet services, typically provided by ISPs, include Internet access,Internet transit, domain name registration, web hosting, andcollocation. ISPs may engage in peering, where multiple ISPsinterconnect at peering points or Internet exchange points (IXs),allowing routing of data between each network, without charging oneanother for the data transmitted—data that would otherwise have passedthrough a third upstream ISP, incurring charges from the upstream ISP.ISPs requiring no upstream and having only customers (end customersand/or peer ISPs) are referred to as Tier 1 ISPs.

An arrangement 10 a of a computer system connected to the Internet 22 isshown in FIG. 1 a . A computer system or a workstation 7 includes a mainunit box 6 with an enclosed motherboard that has the processor 12 andthe memories 15 a, 15 b, and 15 c are mounted. The workstation 7 mayinclude a keyboard 2 (corresponding to the input device 18), a printer4, a computer mouse 3 (corresponding to the cursor control 18 a), and adisplay 5 (corresponding to the display 17). FIG. 1 a furtherillustrates various devices connected via the Internet 22, such as aclient device #1 24, a client device #2 24 a, a data server #1 23 a, adata server #2 23 b, and the workstation 7, connected to the Internet 22over a LAN 14 and via the router or gateway 19 and the ISP 16.

The client device #1 24 and the client device #2 24 a may communicateover the Internet 22 for exchanging or obtaining data from the dataserver #1 23 a and the data server #2 23 b. In one example, the serversare HTTP servers, sometimes known as web servers.

The term “computer-readable medium” (or “machine-readable medium”) isused herein to include, but not limited to, any medium or any memory,that participates in providing instructions to a processor, (such as theprocessor 12) for execution, or any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). Such a medium may store computer-executable instructions tobe executed by a processing element and/or control logic and data, whichis manipulated by a processing element and/or control logic, and maytake many forms, including but not limited to, non-volatile medium,volatile medium, and transmission medium. Transmission media includescoaxial cables, copper wire, and fiber optics, including the wires thatcomprise the bus 13. Transmission media may also take the form ofacoustic or light waves, such as those generated during radio-wave andinfra-red data communications, or other form of propagating signals(e.g., carrier waves, infrared signals, digital signals, etc.). Commonforms of computer-readable media include a floppy disk, a flexible disk,hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, anyother optical medium, punch-cards, paper-tape, any other physical mediumwith patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, anyother memory chip or cartridge, a carrier wave as described hereinafter,or any other medium from which a computer may read.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor 12 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer may load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to the computer system 11can receive the data on the telephone line, using an infraredtransmitter to convert the data to an infrared signal. An infrareddetector can receive the data carried in the infrared signal andappropriate circuitry may place the data on the bus 13. The bus 13carries the data to the main memory 15 a, from which the processor 12retrieves and executes the instructions. The instructions received bythe main memory 15 a may optionally be stored on the storage device 15 ceither before or after execution by the processor 12.

The Internet is a global system of interconnected computer networks thatuse the standardized Internet Protocol Suite (TCP/IP), includingTransmission Control Protocol (TCP) and the Internet Protocol (IP), toserve billions of users worldwide. It is a network of networks thatconsists of millions of private, public, academic, business, andgovernment networks, of local to global scope, that are linked by abroad array of electronic and optical networking technologies. TheInternet carries a vast range of information resources and services,such as the interlinked hypertext documents on the World Wide Web (WWW)and the infrastructure to support electronic mail. The Internet backbonerefers to the principal data routes between large, strategicallyinterconnected networks and core routers on the Internet. These datarouters are hosted by commercial, government, academic, and otherhigh-capacity network centers, the Internet exchange points and networkaccess points that interchange Internet traffic between the countries,continents and across the oceans of the world. Traffic interchangebetween Internet service providers (often Tier 1 networks) participatingin the Internet backbone exchange traffic by privately negotiatedinterconnection agreements, primarily governed by the principle ofsettlement-free peering.

The Internet Protocol is responsible for addressing hosts and routingdatagrams (packets) from a source host to the destination host acrossone or more IP networks. For this purpose, the Internet Protocol definesan addressing system that has two functions: Identifying hosts addressesand providing a logical location service. Each packet is tagged with aheader that contains the metadata for the purpose of delivery. Thisprocess of tagging is also called encapsulation. IP is a connectionlessprotocol for use in a packet-switched Link Layer network, and does notneed circuit setup prior to transmission. The aspects of guaranteeingdelivery, proper sequencing, avoidance of duplicate delivery, and dataintegrity are addressed by an upper transport layer protocol (e.g.,TCP—Transmission Control Protocol and UDP—User Datagram Protocol).

The Hypertext Transfer Protocol (HTTP) is an application protocol fordistributed, collaborative, hypermedia information systems, commonlyused for communication over the Internet. HTTP is the protocol toexchange or transfer hypertext, which is a structured text that useslogical links (hyperlinks) between nodes containing text. HTTP version1.1 was standardized as RFC 2616 (June 1999), which was replaced by aset of standards (obsoleting RFC 2616), including RFC 7230—‘HTTP/1.1:Message Syntax and Routing’, RFC 7231—‘HTTP/1.1: Semantics and Content’,RFC 7232—‘HTTP/1.1: Conditional Requests’, RFC 7233—‘HTTP/1.1: RangeRequests’, RFC 7234—‘HTTP/1.1: Caching’, and RFC 7235—‘HTTP/1.1:Authentication’. HTTP functions as a request-response protocol in theclient-server computing model. A web browser, for example, may be theclient and an application running on a computer hosting a website may bethe server. The client submits an HTTP request message to the server.The server, which provides resources such as HTML files and othercontent, or performs other functions on behalf of the client, returns aresponse message to the client. The response contains completion statusinformation about the request and may further contain a requestedcontent in its message body. A web browser is an example of a User Agent(UA). Other types of user agent include the indexing software used bysearch providers (web crawlers), voice browsers, mobile apps and othersoftware that accesses, consumes, or displays web content.

User. The term “user” is used herein to include, but not limited to, theprincipal using a client device or application to interactively retrieveand render resources or resource manifestation, such as a person using aweb browser, a person using an e-mail reader, or a person using adisplay such as the display 17.

Virtualization. The term virtualization typically refers to thetechnology that allows for the creation of software-based virtualmachines that can run multiple operating systems from a single physicalmachine. In one example, virtual machines can be used to consolidate theworkloads of several under-utilized servers to fewer machines, perhaps asingle machine (server consolidation), providing benefits (perceived orreal, but often cited by vendors) such as savings on hardware,environmental costs, management, and administration of the serverinfrastructure. Virtualization scheme allows for the creation ofsubstitutes for real resources, that is, substitutes that have the samefunctions and external interfaces as their counterparts, but that differin attributes, such as size, performance, and cost. These substitutesare called virtual resources, and their users are typically unaware ofthe substitution.

Virtualization is commonly applied to physical hardware resources bycombining multiple physical resources into shared pools from which usersreceive virtual resources. With virtualization, you can make onephysical resource look like multiple virtual resources. Virtualresources can have functions or features that are not available in theirunderlying physical resources. Virtualization can provide the benefitsof consolidation to reduce hardware cost, such as to efficiently accessand manage resources to reduce operations and systems management costswhile maintaining needed capacity, and to have a single server functionas multiple virtual servers. In addition, virtualization can provideoptimization of workloads, such as to respond dynamically to theapplication needs of its users, and to increase the use of existingresources by enabling dynamic sharing of resource pools. Further,virtualization may be used for IT flexibility and responsiveness, suchas by having a single, consolidated view of, and easy access to, allavailable resources in the network, regardless of location, and reducingthe management of your environment by providing emulation forcompatibility and improved interoperability.

Virtual machine (VM). Virtual machine is a representation of a realmachine using software that provides an operating environment which canrun or host a guest operating system. In one example, a virtual machinemay include a self-contained software emulation of a machine, which doesnot physically exist, but shares resources of an underlying physicalmachine. Like a physical computer, a virtual machine runs an operatingsystem and applications. Multiple virtual machines can operateconcurrently on a single host system. There are different kinds ofvirtual machines, each with different functions: System virtual machines(also termed full virtualization VMs) provide a substitute for a realmachine. They provide functionality needed to execute entire operatingsystems. A hypervisor uses native execution to share and managehardware, allowing for multiple environments which are isolated from oneanother, yet exist on the same physical machine. Modern hypervisors usehardware-assisted virtualization, virtualization-specific hardware,primarily from the host CPUs. Process virtual machines are designed toexecute computer programs in a platform-independent environment. Somevirtual machines, such as QEMU, are designed to also emulate differentarchitectures and allow execution of software applications and operatingsystems written for another CPU or architecture. Operating-system-levelvirtualization allows the resources of a computer to be partitioned viathe kernel's support for multiple isolated user space instances, whichare usually called containers and may look and feel like real machinesto the end users.

Guest Operating System. A guest operating system is an operating systemrunning in a virtual machine environment that would otherwise rundirectly on a separate physical system. Operating-system-levelvirtualization, also known as containerization, refers to an operatingsystem feature in which the kernel allows the existence of multipleisolated user-space instances. Such instances, called containers,partitions, Virtualization Engines (VEs) or jails (FreeBSD jail orchroot jail), may look like real computers from the point of view ofprograms running in them. A computer program running on an ordinaryoperating system can see all resources (connected devices, files andfolders, network shares, CPU power, quantifiable hardware capabilities)of that computer. However, programs running inside a container can onlysee the container's contents and devices assigned to the container. Inaddition to isolation mechanisms, the kernel often providesresource-management features to limit the impact of one container'sactivities on other containers. With operating-system-virtualization, orcontainerization, it is possible to run programs within containers, towhich only parts of these resources are allocated. A program expectingto see the whole computer, once run inside a container, can only see theallocated resources and believes them to be all that is available.Several containers can be created on each operating system, to each ofwhich a subset of the computer's resources is allocated. Each containermay contain any number of computer programs. These programs may runconcurrently or separately, even interact with each other.

Hypervisor. Hypervisor commonly refers to a thin layer of software thatgenerally provides virtual partitioning capabilities which runs directlyon hardware, but underneath higher-level virtualization services. Thehypervisor typically manages virtual machines, allowing them to interactdirectly with the underlying hardware. System virtualization createsmany virtual systems within a single physical system. Virtual systemsare independent operating environments that use virtual resources.System virtualization can be approached through hardware partitioning orhypervisor technology. Hardware partitioning subdivides a physicalserver into fractions, each of which can run an operating system. Thesefractions are typically created with coarse units of allocation, such aswhole processors or physical boards. This type of virtualization allowsfor hardware consolidation, but does not have the full benefits ofresource sharing and emulation offered by hypervisors. Hypervisors use athin layer of code in software or firmware to achieve fine-grained,dynamic resource sharing. Because hypervisors provide the greatest levelof flexibility in how virtual resources are defined and managed, theyare the primary technology for system virtualization.

Virtual Machine Monitor. A Virtual Machine Monitor (VMM) is computersoftware, firmware or hardware that creates and runs virtual machines. Acomputer on which a hypervisor runs one or more virtual machines iscalled a host machine, and each virtual machine is called a guestmachine. The hypervisor presents the guest operating systems with avirtual operating platform and manages the execution of the guestoperating systems. Multiple instances of a variety of operating systemsmay share the virtualized hardware resources: for example, Linux,Windows, and macOS instances can all run on a single physical x86machine. This contrasts with operating-system-level virtualization,where all instances (usually called containers) must share a singlekernel, though the guest operating systems can differ in user space,such as different Linux distributions with the same kernel. Typically, aVMM refers to a software that runs in a layer between a hypervisor orhost operating system and one or more virtual machines that provides thevirtual machines abstraction to the guest operating systems. With fullvirtualization, the VMM exports a virtual machine abstraction identicalto the physical machine, so the standard operating system can run justas they would on physical hardware.

Hardware virtualization or platform virtualization refers to thecreation of a virtual machine that acts like a real computer with anoperating system. Software executed on these virtual machines isseparated from the underlying hardware resources. In hardwarevirtualization, the host machine is the actual machine on which thevirtualization takes place, and the guest machine is the virtualmachine. The words host and guest are used to distinguish the softwarethat runs on the physical machine from the software that runs on thevirtual machine. The software or firmware that creates a virtual machineon the host hardware is called a hypervisor or Virtual Machine Manager.Different types of hardware virtualization include full-virtualization,where almost complete simulation of the actual hardware to allowsoftware, which typically consists of a guest operating system, to rununmodified, and Para-virtualization, where a hardware environment is notsimulated; however, the guest programs are executed in their ownisolated domains, as if they are running on a separate system. Guestprograms need to be specifically modified to run in this environment.

Hardware-assisted virtualization is a way of improving overallefficiency of virtualization. It involves CPUs that provide support forvirtualization in hardware, and other hardware components that helpimprove the performance of a guest environment. Hardware virtualizationcan be viewed as part of an overall trend in enterprise IT that includesautonomic computing, a scenario in which the IT environment will be ableto manage itself based on perceived activity, and utility computing, inwhich computer processing power is seen as a utility that clients canpay for only as needed. The usual goal of virtualization is tocentralize administrative tasks while improving scalability and overallhardware-resource utilization. With virtualization, several operatingsystems can be run in parallel on a single central processing unit(CPU). This parallelism tends to reduce overhead costs and differs frommultitasking, which involves running several programs on the same OS.Using virtualization, an enterprise can better manage updates and rapidchanges to the operating system and applications without disrupting theuser.

Server Virtualization. Server virtualization is a virtualizationtechnique that involves partitioning a physical server into a number ofsmall, virtual servers with the help of virtualization software. Inserver virtualization, each virtual server runs multiple operatingsystem instances at the same time. A Virtual Private Server (VPS) is avirtual machine sold as a service by an Internet hosting service, thatruns its own copy of an Operating System (OS), and customers may havesuperuser-level access to that operating system instance, so they caninstall almost any software that runs on that OS. For many purposes theyare functionally equivalent to a dedicated physical server, and beingsoftware-defined, are able to be much more easily created andconfigured. They are typically priced much lower than an equivalentphysical server. However, as they share the underlying physical hardwarewith other VPS's, performance may be lower, depending on the workload ofany other executing virtual machines. Dedicated Servers may also be moreefficient with CPU dependent processes such as hashing algorithms.

Application Virtualization. Application virtualization is softwaretechnology that encapsulates computer programs from the underlyingoperating system on which it is executed. A fully virtualizedapplication is not installed in the traditional sense, although it isstill executed as if it were. The application behaves at runtime like itis directly interfacing with the original operating system and all theresources managed by it, but can be isolated or sandboxed to varyingdegrees. Application virtualization is layered on top of othervirtualization technologies, allowing computing resources to bedistributed dynamically in real-time. In this context, the term“virtualization” commonly refers to the artifact being encapsulated(application), which is quite different from its meaning in hardwarevirtualization, where it refers to the artifact being abstracted(physical hardware).

Network Virtualization. Network Virtualization refers to the process ofcombining hardware and software network resources to create a singlepool of resources that make up a virtual network that can be accessedwithout regard to the physical component. Network virtualizationtypically involves combining hardware and software network resources andnetwork functionality into a single, software-based administrativeentity, a virtual network. Network virtualization involves platformvirtualization, often combined with resource virtualization. Networkvirtualization is categorized as either external virtualization,combining many networks or parts of networks into a virtual unit, orinternal virtualization, providing network-like functionality tosoftware containers on a single network server.

Storage Virtualization. Storage virtualization refers to the process ofconsolidating the physical storage from multiple network storage devicesso that it appears to be a single storage unit. Within the context of astorage system, there are two primary types of virtualization that canoccur: Block virtualization used in this context refers to theabstraction (separation) of logical storage (partition) from physicalstorage so that it may be accessed without regard to physical storage orheterogeneous structure. This separation allows the administrators ofthe storage system greater flexibility in how they manage storage forend users. File virtualization addresses the NAS challenges byeliminating the dependencies between the data accessed at the file leveland the location where the files are physically stored. This providesopportunities to optimize storage use and server consolidation and toperform non-disruptive file migrations.

Desktop Virtualization. Desktop virtualization refers to the process ofvirtualizing desktop computers using virtualization software, such thatthe desktop computer and the associated operating system andapplications are separated from the physical client device that is usedto access it. Desktop virtualization is software technology thatseparates the desktop environment and associated application softwarefrom the physical client device that is used to access it.

Desktop virtualization can be used in conjunction with applicationvirtualization and user profile management systems, now termed “uservirtualization,” to provide a comprehensive desktop environmentmanagement system. In this mode, all the components of the desktop arevirtualized, which allows for a highly flexible and much more securedesktop delivery model. In addition, this approach supports a morecomplete desktop disaster recovery strategy as all components areessentially saved in the data center and backed up through traditionalredundant maintenance systems. If a user's device or hardware is lost,the restore is straightforward and simple, because the components willbe present at login from another device. In addition, because no data issaved to the user's device, if that device is lost, there is much lesschance that any critical data can be retrieved and compromised. VirtualDesktop Infrastructure (VDI)—The practice of hosting a desktopenvironment within a virtual machine that runs on a centralized orremote server.

An example of a virtualization architecture 900 is shown in FIG. 2 ,where three virtual machines are exemplified. A Virtual Machine (VM) #1910 a provides virtualization for the application 901 a that uses theguest OS 902 a, which in turn interfaces with the virtual hardware 903 athat emulates the actual hardware. Similarly, a Virtual Machine (VM) #2910 b provides virtualization for the application 901 b that uses theguest OS 902 b, which in turn interfaces with the virtual hardware 903 bthat emulates the associated actual hardware, and a Virtual Machine (VM)#3 910 c provides virtualization for the application 901 c that uses theguest OS 902 c, which in turn interfaces with the virtual hardware 903 cthat emulates the associated actual hardware. The abstraction layer isprovided by VMM 904, allowing of hardware-independence of operatingsystem and applications, provisioning on any single physical system, andmanaging the applications and the OSs as a single encapsulated unit.

A hosted architecture 900 a for virtualization is shown in FIG. 2 a ,where a wide range of actual host hardware 906 may be used byimplementing a host operating system 905 layer between the actualhardware 906 and the VMM 904. Such configuration relies on the host OS905 for device support and physical resource management. In contrast, abare-metal architecture 900 b is shown in FIG. 2 b , where a hypervisorlayer (in addition to, or as part of, the VMM 904) is used as the firstlayer, allowing the VMM 904 to have direct access to the hardwareresources, hence providing more efficient, and greater scalability,robustness, and performance.

Cloud. The term “Cloud” or “Cloud computing” as used herein is definedas a technology infrastructure facilitating supplement, consumption anddelivery of IT services, and generally refers to any group of networkedcomputers capable of delivering computing services (such ascomputations, applications, data access, and data management and storageresources) to end users. This disclosure does not limit the type (suchas public or private) of the cloud as well as the underlying systemarchitecture used by the cloud. The IT services are internet based andmay involve elastic provisioning of dynamically scalable and timevirtualized resources. Although such virtualization environments can beprivately deployed and used within local area or wide area networksowned by an enterprise, a number of “cloud service providers” hostvirtualization environments accessible through the public internet (the“public cloud”) that is generally open to anyone, or through private IPor other type of network accessible only by entities given access to it(a “private cloud.”). Using a cloud-based control server or using thesystem above may allow for reduced capital or operational expenditures.The users may further access the system using a web browser regardlessof their location or what device they are using, and the virtualizationtechnology allows servers and storage devices to be shared andutilization be increased. Examples of public cloud providers includeAmazon AWS, Microsoft Azure and Google GCP. Comparison of servicefeatures such as computation, storage, and infrastructure of the threecloud service providers (AWS, Microsoft Azure, GCP) is disclosed in anarticle entitled: “Highlight the Features of AWS, GCP and MicrosoftAzure that Have an Impact when Choosing a Cloud Service Provider” byMuhammad Ayoub Kamal, Hafiz Wahab Raza, Muhammad Mansoor Alam, andMazliham Mohd Su'ud, published January 2020 in ‘International Journal ofRecent Technology and Engineering (IJRTE)’ ISSN: 2277-3878, Volume-8 byBlue Eyes Intelligence Engineering & Sciences Publication[DOI:10.35940/ijrte.D8573.018520], which is incorporated in its entiretyfor all purposes as if fully set forth herein.

The term “Software as a Service (SaaS)” as used herein in thisapplication, is defined as a model of software deployment whereby aprovider licenses a Software Application (SA) to customers for use as aservice on demand. Similarly, an “Infrastructure as a Service” (IaaS)allows enterprises to access virtualized computing systems through thepublic Internet. The term “customer” as used herein in this application,is defined as a business entity that is served by an SA, provided on theSaaS platform. A customer may be a person or an organization and may berepresented by a user that responsible for the administration of theapplication in aspects of permissions configuration, user relatedconfiguration, and data security policy. The service is supplied andconsumed over the Internet, thus eliminating requirements to install andrun applications locally on a site of a customer as well as simplifyingmaintenance and support. Particularly it is advantageous in massivebusiness applications. Licensing is a common form of billing for theservice and it is paid periodically. SaaS is becoming ever more commonas a form of SA delivery over the Internet and is being facilitated in atechnology infrastructure called “Cloud Computing”. In this form of SAdelivery, where the SA is controlled by a service provider, a customermay experience stability and data security issues. In many cases, thecustomer is a business organization that is using the SaaS for businesspurposes such as business software; hence, stability and data securityare primary requirements. As part of a cloud service arrangement, anycomputer system may also be emulated using software running on ahardware computer system. This virtualization allows for multipleinstances of a computer system, each referred to as virtual machine, torun on a single machine. Each virtual machine behaves like a computersystem running directly on hardware. It is isolated from the othervirtual machines, as would two hardware computers. Each virtual machinecomprises an instance of an operating system (the “guest operatingsystem”). There is a host operating system running directly on thehardware that supports the software that emulates the hardware, and theemulation software is referred to as a hypervisor.

The term “cloud-based” generally refers to a hosted service that isremotely located from a data source and configured to receive, store andprocess data delivered by the data source over a network. Cloud-basedsystems may be configured to operate as a public cloud-based service, aprivate cloud-based service or a hybrid cloud-based service. A “publiccloud-based service” may include a third-party provider that suppliesone or more servers to host multi-tenant services. Examples of a publiccloud-based service include Amazon Web Services® (AWS®), Microsoft®Azure™, and Google® Compute Engine™ (GCP) as examples. In contrast, a“private” cloud-based service may include one or more servers that hostservices provided to a single subscriber (enterprise) and a hybridcloud-based service may be a combination of certain functionality from apublic cloud-based service and a private cloud-based service.

Cloud computing and virtualization is described in a book entitled“Cloud Computing and Virtualization” authored by Dac-Nhuong Le (Facultyof Information Technology, Haiphong University, Haiphong, Vietnam),Raghvendra Kumar (Department of Computer Science and Engineering, LNCT,Jabalpur, India), Gia Nhu Nguyen (Graduate School, Duy Tan University,Da Nang, Vietnam), and Jyotir Moy Chatterjee (Department of ComputerScience and Engineering at GD-RCET, Bhilai, India), and published 2018by John Wiley & Sons, Inc. [ISBN 978-1-119-48790-6], which isincorporated in its entirety for all purposes as if fully set forthherein. The book describes the adoption of virtualization in datacenters creates the need for a new class of networks designed to supportelasticity of resource allocation, increasing mobile workloads and theshift to production of virtual workloads, requiring maximumavailability. Building a network that spans both physical servers andvirtual machines with consistent capabilities demands a newarchitectural approach to designing and building the IT infrastructure.Performance, elasticity, and logical addressing structures must beconsidered as well as the management of the physical and virtualnetworking infrastructure. Once deployed, a network that isvirtualization-ready can offer many revolutionary services over a commonshared infrastructure. Virtualization technologies from VMware, Citrixand Microsoft encapsulate existing applications and extract them fromthe physical hardware. Unlike physical machines, virtual machines arerepresented by a portable software image, which can be instantiated onphysical hardware at a moment's notice. With virtualization, comeselasticity where computer capacity can be scaled up or down on demand byadjusting the number of virtual machines actively executing on a givenphysical server. Additionally, virtual machines can be migrated while inservice from one physical server to another.

Extending this further, virtualization creates “location freedom”enabling virtual machines to become portable across an ever-increasinggeographical distance. As cloud architectures and multi-tenancycapabilities continue to develop and mature, there is an economy ofscale that can be realized by aggregating resources across applications,business units, and separate corporations to a common shared, yetsegmented, infrastructure. Elasticity, mobility, automation, and densityof virtual machines demand new network architectures focusing on highperformance, addressing portability, and the innate understanding of thevirtual machine as the new building block of the data center. Consistentnetwork-supported and virtualization-driven policy and controls arenecessary for visibility to virtual machines' state and location as theyare created and moved across a virtualized infrastructure.

Virtualization technologies in data center environments are described ina eBook authored by Gustavo Alessandro Andrade Santana and published2014 by Cisco Systems, Inc. (Cisco Press) [ISBN-13: 978-1-58714-324-3]entitled: “Data Center Virtualization Fundamentals”, which isincorporated in its entirety for all purposes as if fully set forthherein. PowerVM technology for virtualization is described in IBMRedBook entitled: “IBM PowerVM Virtualization—Introduction andConfiguration” published by IBM Corporation June 2013, andvirtualization basics is described in a paper by IBM Corporationpublished 2009 entitled: “Power Systems—Introduction to virtualization”,which are both incorporated in their entirety for all purposes as iffully set forth herein.

Server. The Internet architecture employs a client-server model, amongother arrangements. The terms ‘server’ or ‘server computer’ relatesherein to a device or computer (or a plurality of computers) connectedto the Internet and is used for providing facilities or services toother computers or other devices (referred to in this context as‘clients’) connected to the Internet. A server is commonly a host thathas an IP address and executes a ‘server program’, and typicallyoperates as a socket listener. Many servers have dedicated functionalitysuch as web server, Domain Name System (DNS) server (described in RFC1034 and RFC 1035), Dynamic Host Configuration Protocol (DHCP) server(described in RFC 2131 and RFC 3315), mail server, File TransferProtocol (FTP) server and database server. Similarly, the term ‘client’is used herein to include, but not limited to, a program or to a deviceor a computer (or a series of computers) executing this program, whichaccesses a server over the Internet for a service or a resource. Clientscommonly initiate connections that a server may accept. For non-limitingexample, web browsers are clients that connect to web servers forretrieving web pages, and email clients connect to mail storage serversfor retrieving mails.

A server device (in server/client architecture) typically offersinformation resources, services, and applications to clients, using aserver dedicated or oriented operating system. A server device mayconsist of, be based on, include, or be included in the work-station 7shown in FIG. 2 , the computer system 10 shown in FIG. 1 , or thecomputer 11 shown in FIG. 1 . Current popular server operating systemsare based on Microsoft Windows (by Microsoft Corporation, headquarteredin Redmond, Wash., U.S.A.), Unix, and Linux-based solutions, such as the‘Windows Server 2012’ server operating system, which is a part of theMicrosoft ‘Windows Server’ OS family, that was released by Microsoft in2012. ‘Windows Server 2012’ provides enterprise-class datacenter andhybrid cloud solutions that are simple to deploy, cost-effective,application-specific, and user-centric, and is described in Microsoftpublication entitled: “Inside-Out Windows Server 2012”, by William R.Stanek, published 2013 by Microsoft Press, which is incorporated in itsentirety for all purposes as if fully set forth herein.

Unix operating system is widely used in servers. It is a multitasking,multiuser computer operating system that exists in many variants, and ischaracterized by a modular design that is sometimes called the “Unixphilosophy”, meaning the OS provides a set of simple tools, which eachperforms a limited, well-defined function, with a unified filesystem asthe primary means of communication, and a shell scripting and commandlanguage to combine the tools to perform complex workflows. Unix wasdesigned to be portable, multi-tasking and multi-user in a time-sharingconfiguration, and Unix systems are characterized by various concepts:the use of plain text for storing data, a hierarchical file system,treating devices and certain types of Inter-Process Communication (IPC)as files, the use of a large number of software tools, and smallprograms that can be strung together through a command line interpreterusing pipes, as opposed to using a single monolithic program thatincludes all of the same functionality. Unix operating system consistsof many utilities along with the master control program, the kernel. Thekernel provides services to start and stop programs, handles the filesystem and other common “low level” tasks that most programs share, andschedules access to avoid conflicts when programs try to access the sameresource, or device simultaneously. To mediate such access, the kernelhas special rights, reflected in the division between user-space andkernel-space. Unix is described in a publication entitled: “UNIXTutorial” by tutorialspoint.com, downloaded on July 2014, which isincorporated in its entirety for all purposes as if fully set forthherein.

Client. The term ‘client’ typically refers to an application (or adevice executing the application) used for retrieving or renderingresources, or resource manifestations, such as a web browser, an e-mailreader, or a Usenet reader, while the term ‘server’ typically refers toan application (or a device executing the application) used forsupplying resources or resource manifestations, and typically offers (orhosts) various services to other network computers and users. Theseservices are usually provided through ports or numbered access pointsbeyond the server's network address. Each port number is usuallyassociated with a maximum of one running program, which is responsiblefor handling requests to that port. A daemon, being a user program, canin turn access the local hardware resources of that computer by passingrequests to the operating system kernel.

A client device (in server/client architecture) typically receivesinformation resources, services, and applications from servers, and isusing a client dedicated or oriented operating system. The client devicemay consist of, be based on, include, or be included in, the workstation7, the computer system 10 or the computer 11. Current popular clientoperating systems are based on Microsoft Windows (by MicrosoftCorporation, headquartered in Redmond, Wash., U.S.A.), which is a seriesof graphical interface operating systems developed, marketed, and soldby Microsoft. Microsoft Windows is described in Microsoft publicationsentitled: “Windows Internals—Part 1” and “Windows Internals—Part 2”, byMark Russinovich, David A. Solomon, and Alex Ioescu, published byMicrosoft Press in 2012, which are both incorporated in their entiretyfor all purposes as if fully set forth herein. Windows 8 is a personalcomputer operating system developed by Microsoft as part of Windows NTfamily of operating systems, that was released for general availabilityon October 2012, and is described in Microsoft Press 2012 publicationentitled: “Introducing Windows 8—An Overview for IT Professionals” byJerry Honeycutt, which is incorporated in its entirety for all purposesas if fully set forth herein.

Chrome OS is a Linux kernel-based operating system designed by GoogleInc. out of Mountain View, Calif., U.S.A., to work primarily with webapplications. The user interface takes a minimalist approach andconsists almost entirely of just the Google Chrome web browser; sincethe operating system is aimed at users who spend most of their computertime on the Web, the only “native” applications on Chrome OS are abrowser, media player and file manager, and hence the Chrome OS isalmost a pure web thin client OS.

The Chrome OS is described as including a three-tier architecture:firmware, browser and window manager, and system-level software anduserland services. The firmware contributes to fast boot time by notprobing for hardware, such as floppy disk drives, that are no longercommon on computers, especially netbooks. The firmware also contributesto security by verifying each step in the boot process and incorporatingsystem recovery. The system-level software includes the Linux kernelthat has been patched to improve boot performance. The userland softwarehas been trimmed to essentials, with management by Upstart, which canlaunch services in parallel, re-spawn crashed jobs, and defer servicesin the interest of faster booting. The Chrome OS user guide is describedin the Samsung Electronics Co., Ltd. presentation entitled: “Google™Chrome OS USER GUIDE” published 2011, which is incorporated in itsentirety for all purposes as if fully set forth herein.

RTOS. A Real-Time Operating System (RTOS) is an Operating System (OS)intended to serve real-time applications that process data as it comesin, typically without buffer delays. Processing time requirements(including any OS delay) are typically measured in tenths of seconds orshorter increments of time, and is a time bound system which has welldefined fixed time constraints. Processing is commonly to be done withinthe defined constraints, or the system will fail. They either are eventdriven or time sharing, where event driven systems switch between tasksbased on their priorities while time sharing systems switch the taskbased on clock interrupts. A key characteristic of an RTOS is the levelof its consistency concerning the amount of time it takes to accept andcomplete an application's task; the variability is jitter. A hardreal-time operating system has less jitter than a soft real-timeoperating system. The chief design goal is not high throughput, butrather a guarantee of a soft or hard performance category. An RTOS thatcan usually or generally meet a deadline is a soft real-time OS, but ifit can meet a deadline deterministically it is a hard real-time OS. AnRTOS has an advanced algorithm for scheduling, and includes a schedulerflexibility that enables a wider, computer-system orchestration ofprocess priorities. Key factors in a real-time OS are minimal interruptlatency and minimal thread switching latency; a real-time OS is valuedmore for how quickly or how predictably it can respond than for theamount of work it can perform in a given period of time.

Common designs of RTOS include event-driven, where tasks are switchedonly when an event of higher priority needs servicing; called preemptivepriority, or priority scheduling, and time-sharing, where task areswitched on a regular clocked interrupt, and on events; called roundrobin. Time sharing designs switch tasks more often than strictlyneeded, but give smoother multitasking, giving the illusion that aprocess or user has sole use of a machine. In typical designs, a taskhas three states: Running (executing on the CPU); Ready (ready to beexecuted); and Blocked (waiting for an event, I/O for example). Mosttasks are blocked or ready most of the time because generally only onetask can run at a time per CPU. The number of items in the ready queuecan vary greatly, depending on the number of tasks the system needs toperform and the type of scheduler that the system uses. On simplernon-preemptive but still multitasking systems, a task has to give up itstime on the CPU to other tasks, which can cause the ready queue to havea greater number of overall tasks in the ready to be executed state(resource starvation).

RTOS concepts and implementations are described in an Application NoteNo. RES05B00008-0100/Rec. 1.00 published January 2010 by RenesasTechnology Corp. entitled: “R8C Family—General RTOS Concepts”, in JAJATechnology Review article published February 2007 [1535-5535/$32.00] byThe Association for Laboratory Automation[doi:10.1016/j.jala.2006.10.016] entitled: “An Overview of Real-TimeOperating Systems”, and in Chapter 2 entitled: “Basic Concepts of RealTime Operating Systems” of a book published 2009[ISBN—978-1-4020-9435-4] by Springer Science+Business Media B.V.entitled: “Hardware-Dependent Software—Principles and Practice”, whichare all incorporated in their entirety for all purposes as if fully setforth herein.

QNX. One example of RTOS is QNX, which is a commercial Unix-likereal-time operating system, aimed primarily at the embedded systemsmarket. QNX was one of the first commercially successful microkerneloperating systems and is used in a variety of devices including cars andmobile phones. As a microkernel-based OS, QNX is based on the idea ofrunning most of the operating system kernel in the form of a number ofsmall tasks, known as Resource Managers. In the case of QNX, the use ofa microkernel allows users (developers) to turn off any functionalitythey do not require without having to change the OS itself; instead,those services will simply not run.

FreeRTOS. FreeRTOS™ is a free and open-source Real-Time Operating systemdeveloped by Real Time Engineers Ltd., designed to fit on small embeddedsystems and implements only a very minimalist set of functions: verybasic handle of tasks and memory management, and just sufficient APIconcerning synchronization. Its features include characteristics such aspreemptive tasks, support for multiple microcontroller architectures, asmall footprint (4.3 Kbytes on an ARM7 after compilation), written in C,and compiled with various C compilers. It also allows an unlimitednumber of tasks to run at the same time, and no limitation about theirpriorities as long as used hardware can afford it.

FreeRTOS™ provides methods for multiple threads or tasks, mutexes,semaphores and software timers. A tick-less mode is provided for lowpower applications, and thread priorities are supported. Four schemes ofmemory allocation are provided: allocate only; allocate and free with avery simple, fast, algorithm; a more complex but fast allocate and freealgorithm with memory coalescence; and C library allocate and free withsome mutual exclusion protection. While the emphasis is on compactnessand speed of execution, a command line interface and POSIX-like 10abstraction add-ons are supported. FreeRTOS™ implements multiple threadsby having the host program call a thread tick method at regular shortintervals.

The thread tick method switches tasks depending on priority and around-robin scheduling scheme. The usual interval is 1/1000 of a secondto 1/100 of a second, via an interrupt from a hardware timer, but thisinterval is often changed to suit a particular application. FreeRTOS™ isdescribed in a paper by Nicolas Melot (downloaded July 2015) entitled:“Study of an operating system: FreeRTOS—Operating systems for embeddeddevices”, in a paper (dated Sep. 23, 2013) by Dr. Richard Wall entitled:“Carebot PIC32 MX7ck implementation of Free RTOS”, FreeRTOS™ modules aredescribed in web pages entitled: “FreeRTOS™ Modules” published in thewww.freertos.org website dated 26 Nov. 2006, and FreeRTOS kernel isdescribed in a paper published 1 Apr. 2007 by Rich Goyette of CarletonUniversity as part of ‘SYSC5701: Operating System Methods for Real-TimeApplications’, entitled: “An Analysis and Description of the InnerWorkings of the FreeRTOS Kernel”, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

SafeRTOS. SafeRTOS was constructed as a complementary offering toFreeRTOS, with common functionality but with a uniquely designedsafety-critical implementation. When the FreeRTOS functional model wassubjected to a full HAZOP, weakness with respect to user misuse andhardware failure within the functional model and API were identified andresolved. Both SafeRTOS and FreeRTOS share the same schedulingalgorithm, have similar APIs, and are otherwise very similar, but theywere developed with differing objectives. SafeRTOS was developed solelyin the C language to meet requirements for certification to IEC61508.SafeRTOS is known for its ability to reside solely in the on-chip readonly memory of a microcontroller for standards compliance. Whenimplemented in hardware memory, SafeRTOS code can only be utilized inits original configuration, so certification testing of systems usingthis OS need not re-test this portion of their designs during thefunctional safety certification process.

VxWorks. VxWorks is an RTOS developed as proprietary software anddesigned for use in embedded systems requiring real-time, deterministicperformance and, in many cases, safety and security certification, forindustries, such as aerospace and defense, medical devices, industrialequipment, robotics, energy, transportation, network infrastructure,automotive, and consumer electronics. VxWorks supports Intelarchitecture, POWER architecture, and ARM architectures. The VxWorks maybe used in multicore asymmetric multiprocessing (AMP), symmetricmultiprocessing (SMP), and mixed modes and multi-OS (via Type 1hypervisor) designs on 32- and 64-bit processors. VxWorks comes with thekernel, middleware, board support packages, Wind River Workbenchdevelopment suite and complementary third-party software and hardwaretechnologies. In its latest release, VxWorks 7, the RTOS has beenre-engineered for modularity and upgradeability so the OS kernel isseparate from middleware, applications and other packages. Scalability,security, safety, connectivity, and graphics have been improved toaddress Internet of Things (IoT) needs.

μC/OS. Micro-Controller Operating Systems (MicroC/OS, stylized as μC/OS)is a real-time operating system (RTOS) that is a priority-basedpreemptive real-time kernel for microprocessors, written mostly in theprogramming language C, and is intended for use in embedded systems.MicroC/OS allows defining several functions in C, each of which canexecute as an independent thread or task. Each task runs at a differentpriority, and runs as if it owns the central processing unit (CPU).Lower priority tasks can be preempted by higher priority tasks at anytime. Higher priority tasks use operating system (OS) services (such asa delay or event) to allow lower priority tasks to execute. OS servicesare provided for managing tasks and memory, communicating between tasks,and timing.

In one example, part of, or all of, the steps, methods, or flow chartsdescribed herein are executed (independently or in cooperation) by aclient device, or any device such as the device 35 shown in FIG. 3 .Alternatively or in addition, part of, or all of, the steps, methods, orflow charts described herein are executed (independently or incooperation) by a server device, such as server 23 a shown as part onthe arrangement 30 shown in FIG. 3 . In one example, a client device(such as the device 35) and a server (such as the server 23 a)cooperatively perform part of, or all of, the steps, methods, or flowcharts described herein. For example, lower computing power processor 12may be used in the device 35, since the heavy or resourcefulcomputations are performed at a remote server. Such scheme may obviatethe need for expensive and resourceful device. In another example,memory resources may be saved at the client device by using data storedat a server. The storage 33 in the device 35 may store the Instructions37 a and the Operating System 37 b. The device 35 may mainly be used forinterfacing the user 36, while the major storing and processingresources and activities are provided by the server 23 a.

The output component 34 may include a color display for displayingscreen elements or for organizing on-screen items and controls for dataentry. Further, the device may support the display of split-screenviews. The input component 38 may include dedicated hard controls forfrequently used/accessed functions (e.g., repeat system message). Manysystems used re-configurable keys/buttons whose function changedepending on the application. For example, a switch may be used toactivate the voice recognition system and it may increase systemreliability. The input component 38 and the output component 34 mayfurther cooperate to provide both auditory and visual feedback toconfirm driver inputs and availability of the speech command. Further, astrategy to alert drivers through auditory tones/beeps in advance of thepresentation of information, and/or changes in display status, may beused. This may limit the need for drivers to continuously monitor thesystem, or repeat system messages.

The device 35 may serve as a client device and may access data, such asretrieving data from, or sending data to, the server 23 a over theInternet 22, such as via the ISP 16 as described in FIG. 1 above. Thecommunication with the server 23 a may be via a wireless network 39, byusing the antenna 29 and the wireless transceiver 28 in the device 35.

A diagrammatic representation of a machine in the example form of thecomputing device within which a set of instructions, for causing themachine to perform any one or more of the methods discussed herein, maybe executed. An example of the device 35 that may be used with any ofthe steps, methods, or flow-charts herein is schematically described aspart of an arrangement 30 shown in FIG. 3 . The components in the device35 communicate over a bus 32, which may correspond with the bus 13 inthe computer 11. The computing device 35 may include a mobile phone, asmart phone, a netbook computer, a rackmount server, a router computer,a server computer, a personal computer, a mainframe computer, a laptopcomputer, a tablet computer, a desktop computer etc., within which a setof instructions, for causing the machine to perform any one or more ofthe methods discussed herein, may be executed. In alternativeembodiments, the machine may be connected (e.g., networked) to othermachines in an LAN, an intranet, an extranet, or the Internet. Themachine may operate in the capacity of a server machine in client-servernetwork environment. The machine may be a Personal Computer (PC), aSet-Top Box (STB), a server, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” may alsoinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methods discussed herein.

The device 35 may also include an interface bus for facilitatingcommunication from various interface devices (for example, one or moreoutput components 34, one or more peripheral interfaces, and one or morecommunication components such as the wireless transceiver 28) to thebasic configuration via the bus/interface controller that controls thebus 32. Some of the example output components include a graphicsprocessing unit and an audio processing unit, which may be configured tocommunicate to various external devices such as a display or speakersvia one or more A/V ports. One or more example peripheral interfaces mayinclude a serial interface controller or a parallel interfacecontroller, which may be configured to communicate with external devicessuch as input components (for example, keyboard, mouse, pen, voice inputdevice, touch input device, etc.) or other peripheral output devices(for example, printer, scanner, etc.) via one or more I/O ports.

The device 35 may be part of, may include, or may be integrated with, ageneral purpose computing device, arranged in accordance with at leastsome embodiments described herein. In an example basic configuration,the device 35 may include one or more processors 12 and one or morememories or any other computer readable media. A dedicated memory busmay be used to communicate between the processor 12 and the devicememories, such as the ROM 15 b, the main memory 15 a, and a storage 33.Depending on the desired configuration, the processor 12 may be of anytype, including but not limited to a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination thereof. The processor 12 may include one or more levels ofcaching, such as a cache memory, a processor core, and registers. Theexample processor core may include an Arithmetic Logic Unit (ALU), aFloating Point Unit (FPU), a Digital Signal Processing core (DSP Core),or any combination thereof. An example memory controller may also beused with the processor 12, or in some implementations, the memorycontroller may be an internal part of the processor 12.

Depending on the desired configuration, the device memories may be ofany type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. The storage 33 may correspond to the storage device 15 c, andmay be part of, may comprise, or may be integrated with the ROM 15 b andthe main memory 15 a. The storage 33 may include an operating system 37c, instruction set 37 b that may include steps or part of, or whole of,the flow-charts described herein. The storage 33 may further include acontrol module, and program data, which may include path data. Any ofthe memories or storages of the device 35 may include read-only memory(ROM), such as ROM 15 b, flash memory, Dynamic Random Access Memory(DRAM) such as Synchronous DRAM (SDRAM)), a static memory (e.g., flashmemory, Static Random Access Memory (SRAM)) and a data storage device,which communicate with each other via the bus 32.

The device 35 may have additional features or functionality, andadditional interfaces to facilitate communications between the basicconfiguration shown in FIG. 3 and any desired devices and interfaces.For example, a bus/interface controller may be used to facilitatecommunications between the basic configuration and one or more datastorage devices via a storage interface bus. The data storage devicesmay be one or more removable storage devices, one or more non-removablestorage devices, or a combination thereof. Examples of the removablestorage and the non-removable storage devices include magnetic diskdevices such as flexible disk drives and hard-disk drives (HDDs),optical disk drives such as compact disk (CD) drives or digitalversatile disk (DVD) drives, solid state drives (SSDs), and tape drivesto name a few. Example computer storage media may include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information, such as computer readableinstructions, data structures, program modules, or other data.

The device 35 may receive inputs from a user 36 via an input component38, which may correspond with the input device 18 or cursor control 18 ashown as part of the computer 11 in FIG. 1 , or may correspond with thepointing device 3 or the keyboard 2 shown as part of the computer system7 in FIG. 1 a . In one example, the input component 38 may be used forreceiving instructions from the user 36. The device 35 notifies oroutputs information to the user 36 using an output component 34, whichmay correspond to the display 17 shown as part of the computer 11 inFIG. 1 , or may correspond with the printer 4 or the screen 5 shown aspart of the computer system 7 in FIG. 1 a . In one example, the outputcomponent 34 may be used for displaying guidance to the user 36.

The interface with the user 36 may be based on the input component 38and the output component 34. For example, receiving input (visually oracoustically) from the user 36 via the input component 38. Similarly,outputting data (visually or acoustically) to the user 36 via the outputcomponent 34. The input component 38 may be a piece of computer hardwareequipment used to provide data and control signals to an informationprocessing system such as a computer or information appliance. Suchinput component 38 may be an integrated or a peripheral input device(e.g., hard/soft keyboard, mouse, resistive or capacitive touch display,etc.). Examples of input components include keyboards, mouse, scanners,digital cameras and joysticks. Input components 38 can be categorizedbased on the modality of input (e.g., mechanical motion, audio, visual,etc.), whether the input is discrete (e.g. pressing of key) orcontinuous (e.g., a mouse's position, though digitized into a discretequantity, is fast enough to be considered continuous), the number ofdegrees of freedom involved (e.g., two-dimensional traditional mice, orthree-dimensional navigators designed for CAD applications). Pointingdevices (such as ‘computer mouse’), which are input components used tospecify a position in space, can further be classified according towhether the input is direct or indirect. With direct input, the inputspace coincides with the display space, i.e., pointing is done in thespace where visual feedback or the pointer appears. Touchscreens andlight pens involve direct input. Examples involving indirect inputinclude the mouse and trackball, and whether the positional informationis absolute (e.g., on a touch screen) or relative (e.g., with a mousethat can be lifted and repositioned). Direct input is almost necessarilyabsolute, but indirect input may be either absolute or relative. Forexample, digitizing graphics tablets that do not have an embedded screeninvolve indirect input and sense absolute positions and are often run inan absolute input mode, but they may also be set up to simulate arelative input mode like that of a touchpad, where the stylus or puckcan be lifted and repositioned.

In the case of wireless networking, the wireless network 39 may use anytype of modulation, such as Amplitude Modulation (AM), a FrequencyModulation (FM), or a Phase Modulation (PM). Further, the wirelessnetwork 39 may be a control network (such as ZigBee or Z-Wave), a homenetwork, a WPAN (Wireless Personal Area Network), a WLAN (wireless LocalArea Network), a WWAN (Wireless Wide Area Network), or a cellularnetwork. An example of a Bluetooth-based wireless controller that may beincluded in a wireless transceiver is SPBT2632C1A Bluetooth moduleavailable from STMicroelectronics NV and described in the data sheetDocID022930 Rev. 6 dated April 2015 entitled: “SPBT2632C1A—Bluetooth®technology class-1 module”, which is incorporated in its entirety forall purposes as if fully set forth herein.

Some embodiments may be used in conjunction with one or more types ofwireless communication signals and/or systems, for example, RadioFrequency (RF), Infra-Red (IR), Frequency-Division Multiplexing (FDM),Orthogonal FDM (OFDM), Time-Division Multiplexing (TDM), Time-DivisionMultiple Access (TDMA), Extended TDMA (E-TDMA), General Packet RadioService (GPRS), extended GPRS, Code-Division Multiple Access (CDMA),Wideband CDMA (WCDMA), CDMA 2000, single-carrier CDMA, multi-carrierCDMA, Multi-Carrier Modulation (MDM), Discrete Multi-Tone (DMT),Bluetooth®, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBee™,Ultra-Wideband (UWB), Global System for Mobile communication (GSM), 2G,2.5G, 3G, 3.5G, Enhanced Data rates for GSM Evolution (EDGE), or thelike. Further, a wireless communication may be based on, or may becompatible with, wireless technologies that are described in Chapter 20:“Wireless Technologies” of the publication number 1-587005-001-3 byCisco Systems, Inc. (7/99) entitled: “Internetworking TechnologiesHandbook”, which is incorporated in its entirety for all purposes as iffully set forth herein.

Alternatively or in addition, the networking or the communication withthe of the wireless-capable device 35 with the server 23 a over thewireless network 39 may be using, may be according to, may be compatiblewith, or may be based on, Near Field Communication (NFC) using passiveor active communication mode, and may use the 13.56 MHz frequency band,and data rate may be 106 Kb/s, 212 Kb/s, or 424 Kb/s, and the modulationmay be Amplitude-Shift-Keying (ASK), and may be according to, may becompatible with, or based on, ISO/IEC 18092, ECMA-340, ISO/IEC 21481, orECMA-352. In such a case, the wireless transceiver 28 may be an NFCtransceiver and the respective antenna 29 may be an NFC antenna.

Alternatively or in addition, the networking or the communication withthe of the wireless-capable device 35 with the server 23 a over thewireless network 39 may be using, may be according to, may be compatiblewith, or may be based on, a Wireless Personal Area Network (WPAN) thatmay be according to, may be compatible with, or based on, Bluetooth™ orIEEE 802.15.1-2005 standards, and the wireless transceiver 28 may be aWPAN modem, and the respective antenna 29 may be a WPAN antenna. TheWPAN may be a wireless control network according to, may be compatiblewith, or based on, ZigBee™ or Z-Wave™ standards, such as IEEE802.15.4-2003.

Alternatively or in addition, the networking or the communication withthe of the wireless-capable device 35 with the server 23 a over thewireless network 39 may be using, may be according to, may be compatiblewith, or may be based on, a Wireless Local Area Network (WLAN) that maybe according to, may be compatible with, or based on, IEEE 802.11a, IEEE802.11b, IEEE 802.11g, IEEE 802.11n, or IEEE 802.11ac standards, and thewireless transceiver 28 may be a WLAN modem, and the respective antenna29 may be a WLAN antenna.

Alternatively or in addition, the networking or the communication withthe of the wireless-capable device 35 with the server 23 a over thewireless network 39 may be using, may be according to, may be compatiblewith, or may be based on, a wireless broadband network or a WirelessWide Area Network (WWAN), and the wireless transceiver 28 may be a WWANmodem, and the respective antenna 29 may be a WWAN antenna. The WWAN maybe a WiMAX network such as according to, may be compatible with, orbased on, IEEE 802.16-2009, and the wireless transceiver 28 may be aWiMAX modem, and the respective antenna 29 may be a WiMAX antenna.Alternatively or in addition, the WWAN may be a cellular telephonenetwork and the wireless transceiver 28 may be a cellular modem, and therespective antenna 29 may be a cellular antenna. The WWAN may be a ThirdGeneration (3G) network and may use UMTS W-CDMA, UMTS HSPA, UMTS TDD,CDMA2000 1×RTT, CDMA2000 EV-DO, or GSM EDGE-Evolution. The cellulartelephone network may be a Fourth Generation (4G) network and may useHSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be based on, or maybe compatible with, IEEE 802.20-2008. Alternatively or in addition, theWWAN may be a satellite network, and the wireless transceiver 28 may bea satellite modem, and the respective antenna 29 may be a satelliteantenna.

Alternatively or in addition, the networking or the communication withthe of the wireless-capable device 35 with the server 23 a over thewireless network 39 may be using, may be according to, may be compatiblewith, or may be based on, a licensed or an unlicensed radio frequencyband, such as the Industrial, Scientific and Medical (ISM) radio band.For example, an unlicensed radio frequency band may be used that may beabout 60 GHz, may be based on beamforming, and may support a data rateof above 7 Gb/s, such as according to, may be compatible with, or basedon, WiGig™, IEEE 802.11ad, WirelessHD™ or IEEE 802.15.3c-2009, and maybe operative to carry uncompressed video data, and may be according to,may be compatible with, or based on, WHDI™ Alternatively or in addition,the wireless network may use a white space spectrum that may be ananalog television channel consisting of a 6 MHz, 7 MHz or 8 MHzfrequency band, and allocated in the 54-806 MHz band. The wirelessnetwork may be operative for channel bonding, and may use two or moreanalog television channels, and may be based on Wireless Regional AreaNetwork (WRAN) standard using OFDMA modulation. Further, the wirelesscommunication may be based on geographically-based cognitive radio, andmay be according to, may be compatible with, or based on, IEEE 802.22 orIEEE 802.11af standards. Real-Time Clock (RTC) ICs measure time evenwhen the power of the main device is off. During these times, RTC ICsdraw power from an auxiliary battery or supercapacitor. Most modern RTCICs reduce package pin count by supporting a serial interface. Anexample of an RTC IC is model No. DS1339A available from MaximIntegrated Products, Inc. (Headquartered in San Jose, Calif., U.S.A.),described in a data sheet No. 19-6425; Rev 2; 1/15 (2015) by MaximIntegrated Products, Inc. entitled: “DS1339A—Low-Current, 12C, SerialReal-Time Clock”, which is incorporated in its entirety for all purposesas if fully set forth herein, and may be used as described in a tutorial5791 (dated Mar. 28, 2014) by Maxim Integrated Products, Inc. entitled:“Tips for Writing Bulletproof Real-Time Clock Control Code”, which isincorporated in its entirety for all purposes as if fully set forthherein.

Smartphone. A mobile phone (also known as a cellular phone, cell phone,smartphone, or hand phone) is a device which can make and receivetelephone calls over a radio link whilst moving around a wide geographicarea, by connecting to a cellular network provided by a mobile networkoperator. The calls are to and from the public telephone network, whichincludes other mobiles and fixed-line phones across the world. TheSmartphones are typically hand-held and may combine the functions of apersonal digital assistant (PDA), and may serve as portable mediaplayers and camera phones with high-resolution touch-screens, webbrowsers that can access, and properly display, standard web pagesrather than just mobile-optimized sites, GPS navigation, Wi-Fi, andmobile broadband access. In addition to telephony, the Smartphones maysupport a wide variety of other services such as text messaging, MMS,email, Internet access, short-range wireless communications (infrared,Bluetooth), business applications, gaming and photography.

An example of a contemporary smartphone is model iPhone 12 Pro Maxavailable from Apple Inc., headquartered in Cupertino, Calif., U.S.A.and described in iPhone 12 Pro Max technical specification and in aweb-page by Apple Inc. entitled: “About iOS 14 Updates” (both retrievedNovember 2020 from www.apple.com), which are both incorporated in theirentirety for all purposes as if fully set forth herein. Another exampleof a smartphone is Samsung Galaxy S20 available from Samsung Electronicsheadquartered in Suwon, South-Korea, described in a document numberUNL_STR_G981U_G986U_G988U_EN_UM_TN_TAW_021220_FINAL entitled: “GalaxyS20/S20+/S20 Untra5G—User manual” (retrieved 11/2020 fromwww.samsung.com), which is incorporated in its entirety for all purposesas if fully set forth herein.

Android is an open source and Linux-based mobile operating system (OS)based on the Linux kernel that is currently offered by Google. With auser interface based on direct manipulation, Android is designedprimarily for touchscreen mobile devices such as smartphones and tabletcomputers, with specialized user interfaces for televisions (AndroidTV), cars (Android Auto), and wrist watches (Android Wear). The OS usestouch inputs that loosely correspond to real-world actions, such asswiping, tapping, pinching, and reverse pinching to manipulate on-screenobjects, and a virtual keyboard. Despite being primarily designed fortouchscreen input, it also has been used in game consoles, digitalcameras, and other electronics. The response to user input is designedto be immediate and provides a fluid touch interface, often using thevibration capabilities of the device to provide haptic feedback to theuser. Internal hardware such as accelerometers, gyroscopes and proximitysensors are used by some applications to respond to additional useractions, for example, adjusting the screen from portrait to landscapedepending on how the device is oriented, or allowing the user to steer avehicle in a racing game by rotating the device by simulating control ofa steering wheel.

Android devices boot to the homescreen, the primary navigation andinformation point on the device, which is similar to the desktop foundon PCs. Android homescreens are typically made up of app icons andwidgets; app icons launch the associated app, whereas widgets displaylive, auto-updating content such as the weather forecast, the user'semail inbox, or a news ticker directly on the homescreen. A homescreenmay be made up of several pages that the user can swipe back and forthbetween, though Android's homescreen interface is heavily customizable,allowing the user to adjust the look and feel of the device to theirtastes. Third-party apps available on Google Play and other app storescan extensively re-theme the homescreen, and even mimic the look ofother operating systems, such as Windows Phone. The Android OS isdescribed in a publication entitled: “Android Tutorial”, downloaded fromtutorialspoint.com on July 2014, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

iOS (previously iPhone OS) from Apple Inc. (headquartered in Cupertino,Calif., U.S.A.) is a mobile operating system distributed exclusively forApple hardware. The user interface of the iOS is based on the concept ofdirect manipulation, using multi-touch gestures. Interface controlelements consist of sliders, switches, and buttons. Interaction with theOS includes gestures such as swipe, tap, pinch, and reverse pinch, allof which have specific definitions within the context of the iOSoperating system and its multi-touch interface. Internal accelerometersare used by some applications to respond to shaking the device (onecommon result is the undo command) or rotating it in three dimensions(one common result is switching from portrait to landscape mode). TheiOS OS is described in a publication entitled: “IOS Tutorial”,downloaded from tutorialspoint.com on July 2014, which is incorporatedin its entirety for all purposes as if fully set forth herein.

Tablet. A tablet computer, commonly referred to as ‘tablet’, is a mobiledevice, typically with a mobile operating system and touchscreen displayprocessing circuitry, and a rechargeable battery in a single, thin andflat package. Modern tablets largely resemble modern smartphones and areused for personal, educational and workplace applications, and the onlydifferences being that tablets are relatively larger than smartphones,with screens 7 inches (18 cm) or larger, measured diagonally, and maynot support access to a cellular network. The touchscreen display istypically operated by gestures executed by finger or digital pen(stylus), instead of the mouse, trackpad, and keyboard of largercomputers. Portable computers can be classified according to thepresence and appearance of physical keyboards. Two species of tablet,the slate and booklet, do not have physical keyboards and usually accepttext and other input by use of a virtual keyboard shown on theirtouchscreen displays. To compensate for their lack of a physicalkeyboard, most tablets can connect to independent physical keyboards byBluetooth or USB.

The size of a slate shaped tablets varies, but commonly slates begin at6 inches (approximately 15 cm). Some models in the larger than 10-inch(25 cm). Mini tablets are smaller and weigh less than slates, withtypical screen sizes between 7-8 inches (18-20 cm). Smartphones andtablets are similar devices, differentiated by the former typicallyhaving smaller screens and most tablets lacking cellular networkcapability.

Two major architectures dominate the tablet market, ARM Holdings' ARMarchitecture and Intel's and AMD's x86. A key component among tabletcomputers is touch input on a touchscreen display. This allows the userto navigate easily and type with a virtual keyboard on the screen orpress other icons on the screen to open apps or files. The system mustrespond to on-screen touches rather than clicks of a keyboard or mouse.This operation makes precise use of our eye—hand coordination.Touchscreens usually come in one of two forms—resistive and capacitive.Resistive touchscreens are passive and respond to pressure on thescreen. They allow a high level of precision, useful in emulating apointer (as is common in tablet computers) but may require calibration.Because of the high resolution, a stylus or fingernail is often used.Stylus-oriented systems are less suited to multi-touch. Capacitivetouchscreens tend to be less accurate, but more responsive thanresistive devices. Because they require a conductive material, such as afingertip, for input, they are not common among stylus-oriented devicesbut are prominent on consumer devices. Most finger-driven capacitivescreens do not currently support pressure input, but some tablets use apressure-sensitive stylus or active pen. Some tablets can recognizeindividual palms, while some professional-grade tablets usepressure-sensitive films, such as those on graphics tablets. Somecapacitive touch-screens can detect the size of the touched area and thepressure used.

Operating system. A mobile operating system (also referred to as mobileOS), is an operating system that operates a smartphone, tablet, PDA, oranother mobile device. Modern mobile operating systems combine thefeatures of a personal computer operating system with other features,including a touchscreen, cellular, Bluetooth, Wi-Fi, GPS mobilenavigation, camera, video camera, speech recognition, voice recorder,music player, near field communication and infrared blaster. Currently,the popular mobile OSs include Android, Symbian, Apple iOS, BlackBerry,MeeGo, Windows Phone, and Bada. Mobile devices with mobilecommunications capabilities (e.g. smartphones) typically contain twomobile operating systems: a main user-facing software platform issupplemented by a second low-level proprietary real-time operatingsystem that operates the radio and other hardware.

Android is a Linux-based, open source mobile operating system (OS) basedon the Linux kernel that is currently offered by Google. With a userinterface based on direct manipulation, Android is designed primarilyfor touchscreen mobile devices such as smartphones and tablet computerswith specialized user interfaces for televisions (Android TV), cars(Android Auto), and wrist watches (Android Wear). The OS uses touchinputs that loosely correspond to real-world actions, such as swiping,tapping, pinching, and reverse pinching to manipulate on-screen objects,and a virtual keyboard. Despite being primarily designed for touchscreeninput, it also has been used in game consoles, digital cameras, andother electronics. The response to user input is designed to beimmediate and provides a fluid touch interface, often using thevibration capabilities of the device to provide haptic feedback to theuser. Internal hardware such as accelerometers, gyroscopes and proximitysensors are used by some applications to respond to additional useractions. For example, adjusting the screen from portrait to landscapedepending on the device orientation, or allowing the user to steer avehicle in a racing game by rotating the device, a process thatsimulates control of a steering wheel.

Android devices boot to the homescreen, the primary navigation andinformation point on the device, which is similar to the desktop foundon PCs. The homescreens on Android are typically made up of app iconsand widgets. App icons launch the associated app, whereas widgetsdisplay live, auto-updating content such as the weather forecast, theuser's email inbox, or a news ticker directly on the homescreen. Ahomescreen may be made up of several pages that the user can swipe backand forth between pages. A heavily-customizable Android homescreeninterface allows the user to adjust the look and feel of the device totheir liking. Third-party apps available on Google Play and other appstores can extensively re-theme the homescreen, and even mimic the lookof other operating systems, such as Windows Phone. The Android OS isdescribed in a publication entitled: “Android Tutorial”, downloaded fromtutorialspoint.com on July 2014, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

iOS (previously iPhone OS) from Apple Inc. (headquartered in Cupertino,Calif., U.S.A.) is a mobile operating system distributed exclusively forApple hardware. The user interface of the iOS is based on the concept ofdirect manipulation, using multi-touch gestures. Interface controlelements consist of sliders, switches, and buttons. Interaction with theOS includes gestures such as swipe, tap, pinch, and reverse pinch, allof which have specific definitions within the context of the iOSoperating system and its multi-touch interface. Internal accelerometersare used by some applications to respond to shaking the device (onecommon result is the undo command), or rotating it in three dimensions(one common result is switching from portrait to landscape mode). TheiOS is described in a publication entitled: “IOS Tutorial”, downloadedfrom tutorialspoint.com on July 2014, which is incorporated in itsentirety for all purposes as if fully set forth herein.

Wireless. Any embodiment herein may be used in conjunction with one ormore types of wireless communication signals and/or systems, forexample, Radio Frequency (RF), Infra-Red (IR), Frequency-DivisionMultiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing(TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA),General Packet Radio Service (GPRS), extended GPRS, Code-DivisionMultiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, single-carrierCDMA, multi-carrier CDMA, Multi-Carrier Modulation (MDM), DiscreteMulti-Tone (DMT), Bluetooth®, Global Positioning System (GPS), Wi-Fi,Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobilecommunication (GSM), Second Generation (2G), 2.5G, Third Generation(3G), 3.5G, Enhanced Data rates for GSM Evolution (EDGE), FourthGeneration (4G), Fifth Generation (5G), or the like. Any wirelessnetwork or wireless connection herein may be operating substantially inaccordance with existing IEEE 802.11, 802.11a, 802.11b, 802.11g,802.11k, 802.11n, 802.11r, 802.16, 802.16d, 802.16e, 802.20, 802.21standards and/or future versions and/or derivatives of the abovestandards. Further, a network element (or a device) herein may consistof, be part of, or include, a cellular radio-telephone communicationsystem, a cellular telephone, a wireless telephone, a PersonalCommunication Systems (PCS) device, a PDA device that incorporates awireless communication device, or a mobile/portable Global PositioningSystem (GPS) device. Further, a wireless communication may be based onwireless technologies that are described in Chapter 20: “WirelessTechnologies” of the publication number 1-587005-001-3 by Cisco Systems,Inc. (7/99) entitled: “Internetworking Technologies Handbook”, which isincorporated in its entirety for all purposes as if fully set forthherein. Wireless technologies and networks are further described in abook published 2005 by Pearson Education, Inc. William Stallings [ISBN:0-13-191835-4] entitled: “Wireless Communications and Networks—secondEdition”, which is incorporated in its entirety for all purposes as iffully set forth herein.

Wireless networking typically employs an antenna (a.k.a. aerial), whichis an electrical device that converts electric power into radio waves,and vice versa, connected to a wireless radio transceiver. Intransmission, a radio transmitter supplies an electric currentoscillating at radio frequency to the antenna terminals, and the antennaradiates the energy from the current as electromagnetic waves (radiowaves). In reception, an antenna intercepts some of the power of anelectromagnetic wave in order to produce a low voltage at its terminalsthat is applied to a receiver to be amplified. Typically an antennaconsists of an arrangement of metallic conductors (elements),electrically connected (often through a transmission line) to thereceiver or transmitter. An oscillating current of electrons forcedthrough the antenna by a transmitter will create an oscillating magneticfield around the antenna elements, while the charge of the electronsalso creates an oscillating electric field along the elements. Thesetime-varying fields radiate away from the antenna into space as a movingtransverse electromagnetic field wave. Conversely, during reception, theoscillating electric and magnetic fields of an incoming radio wave exertforce on the electrons in the antenna elements, causing them to moveback and forth, creating oscillating currents in the antenna. Antennascan be designed to transmit and receive radio waves in all horizontaldirections equally (omnidirectional antennas), or preferentially in aparticular direction (directional or high gain antennas). In the lattercase, an antenna may also include additional elements or surfaces withno electrical connection to the transmitter or receiver, such asparasitic elements, parabolic reflectors or horns, which serve to directthe radio waves into a beam or other desired radiation pattern.

ISM. The Industrial, Scientific and Medical (ISM) radio bands are radiobands (portions of the radio spectrum) reserved internationally for theuse of radio frequency (RF) energy for industrial, scientific andmedical purposes other than telecommunications. In general,communications equipment operating in these bands must tolerate anyinterference generated by ISM equipment, and users have no regulatoryprotection from ISM device operation. The ISM bands are defined by theITU-R in 5.138, 5.150, and 5.280 of the Radio Regulations. Individualcountries use of the bands designated in these sections may differ dueto variations in national radio regulations. Because communicationdevices using the ISM bands must tolerate any interference from ISMequipment, unlicensed operations are typically permitted to use thesebands, since unlicensed operation typically needs to be tolerant ofinterference from other devices anyway. The ISM bands share allocationswith unlicensed and licensed operations; however, due to the highlikelihood of harmful interference, licensed use of the bands istypically low. In the United States, uses of the ISM bands are governedby Part 18 of the Federal Communications Commission (FCC) rules, whilePart 15 contains the rules for unlicensed communication devices, eventhose that share ISM frequencies. In Europe, the MI is responsible forgoverning ISM bands.

Commonly used ISM bands include a 2.45 GHz band (also known as 2.4 GHzband) that includes the frequency band between 2.400 GHz and 2.500 GHz,a 5.8 GHz band that includes the frequency band 5.725-5.875 GHz, a 24GHz band that includes the frequency band 24.000-24.250 GHz, a 61 GHzband that includes the frequency band 61.000-61.500 GHz, a 122 GHz bandthat includes the frequency band 122.000-123.000 GHz, and a 244 GHz bandthat includes the frequency band 244.000-246.000 GHz.

ZigBee. ZigBee is a standard for a suite of high-level communicationprotocols using small, low-power digital radios based on an IEEE 802standard for Personal Area Network (PAN). Applications include wirelesslight switches, electrical meters with in-home-displays, and otherconsumer and industrial equipment that require a short-range wirelesstransfer of data at relatively low rates. The technology defined by theZigBee specification is intended to be simpler and less expensive thanother WPANs, such as Bluetooth. ZigBee is targeted at Radio-Frequency(RF) applications that require a low data rate, long battery life, andsecure networking. ZigBee has a defined rate of 250 kbps suited forperiodic or intermittent data or a single signal transmission from asensor or input device.

ZigBee builds upon the physical layer and medium access control definedin IEEE standard 802.15.4 (2003 version) for low-rate WPANs. Thespecification further discloses four main components: network layer,application layer, ZigBee Device Objects (ZDOs), andmanufacturer-defined application objects, which allow for customizationand favor total integration. The ZDOs are responsible for a number oftasks, which include keeping of device roles, management of requests tojoin a network, device discovery, and security. Because ZigBee nodes cango from a sleep to active mode in 30 ms or less, the latency can be lowand devices can be responsive, particularly compared to Bluetoothwake-up delays, which are typically around three seconds. ZigBee nodescan sleep most of the time, thus the average power consumption can belower, resulting in longer battery life.

There are three defined types of ZigBee devices: ZigBee Coordinator(ZC), ZigBee Router (ZR), and ZigBee End Device (ZED). ZigBeeCoordinator (ZC) is the most capable device and forms the root of thenetwork tree and might bridge to other networks. There is exactly onedefined ZigBee coordinator in each network, since it is the device thatstarted the network originally. It is able to store information aboutthe network, including acting as the Trust Center & repository forsecurity keys. ZigBee Router (ZR) may be running an application functionas well as may be acting as an intermediate router, passing on data fromother devices. ZigBee End Device (ZED) contains functionality to talk toa parent node (either the coordinator or a router). This relationshipallows the node to be asleep a significant amount of the time, therebygiving long battery life. A ZED requires the least amount of memory, andtherefore can be less expensive to manufacture than a ZR or ZC.

The protocols build on recent algorithmic research (Ad-hoc On-demandDistance Vector, neuRFon) to automatically construct a low-speed ad-hocnetwork of nodes. In most large network instances, the network will be acluster of clusters. It can also form a mesh or a single cluster. Thecurrent ZigBee protocols support beacon and non-beacon enabled networks.In non-beacon-enabled networks, an unslotted CSMA/CA channel accessmechanism is used. In this type of network, ZigBee Routers typicallyhave their receivers continuously active, requiring a more robust powersupply. However, this allows for heterogeneous networks in which somedevices receive continuously, while others only transmit when anexternal stimulus is detected.

In beacon-enabled networks, the special network nodes called ZigBeeRouters transmit periodic beacons to confirm their presence to othernetwork nodes. Nodes may sleep between the beacons, thus lowering theirduty cycle and extending their battery life. Beacon intervals depend onthe data rate; they may range from 15.36 milliseconds to 251.65824seconds at 250 Kbit/s, from 24 milliseconds to 393.216 seconds at 40Kbit/s, and from 48 milliseconds to 786.432 seconds at 20 Kbit/s. Ingeneral, the ZigBee protocols minimize the time the radio is on toreduce power consumption. In beaconing networks, nodes only need to beactive while a beacon is being transmitted. In non-beacon-enablednetworks, power consumption is decidedly asymmetrical: some devices arealways active while others spend most of their time sleeping.

Except for the Smart Energy Profile 2.0, current ZigBee devices conformto the IEEE 802.15.4-2003 Low-Rate Wireless Personal Area Network(LR-WPAN) standard. The standard specifies the lower protocol layers—thePHYsical layer (PHY), and the Media Access Control (MAC) portion of theData Link Layer (DLL). The basic channel access mode is “Carrier Sense,Multiple Access/Collision Avoidance” (CSMA/CA), that is, the nodes talkin the same way that people converse; they briefly check to see that noone is talking before they start. There are three notable exceptions tothe use of CSMA. Beacons are sent on a fixed time schedule, and do notuse CSMA. Message acknowledgments also do not use CSMA. Finally, devicesin Beacon Oriented networks that have low latency real-time requirement,may also use Guaranteed Time Slots (GTS), which by definition do not useCSMA.

Z-Wave. Z-Wave is a wireless communications protocol by the Z-WaveAlliance (http://www.z-wave.com) designed for home automation,specifically for remote control applications in residential and lightcommercial environments. The technology uses a low-power RF radioembedded or retrofitted into home electronics devices and systems, suchas lighting, home access control, entertainment systems and householdappliances. Z-Wave communicates using a low-power wireless technologydesigned specifically for remote control applications. Z-Wave operatesin the sub-gigahertz frequency range, around 900 MHz. This band competeswith some cordless telephones and other consumer electronics devices,but avoids interference with WiFi and other systems that operate on thecrowded 2.4 GHz band. Z-Wave is designed to be easily embedded inconsumer electronics products, including battery-operated devices suchas remote controls, smoke alarms, and security sensors.

Z-Wave is a mesh networking technology where each node or device on thenetwork is capable of sending and receiving control commands throughwalls or floors, and use intermediate nodes to route around householdobstacles or radio dead spots that might occur in the home. Z-Wavedevices can work individually or in groups, and can be programmed intoscenes or events that trigger multiple devices, either automatically orvia remote control. The Z-wave radio specifications include bandwidth of9,600 bit/s or 40 Kbit/s, fully interoperable, GFSK modulation, and arange of approximately 100 feet (or 30 meters) assuming “open air”conditions, with reduced range indoors depending on building materials,etc. The Z-Wave radio uses the 900 MHz ISM band: 908.42 MHz (UnitedStates); 868.42 MHz (Europe); 919.82 MHz (Hong Kong); and 921.42 MHz(Australia/New Zealand).

Z-Wave uses a source-routed mesh network topology and has one or moremaster controllers that control routing and security. The devices cancommunicate to another by using intermediate nodes to actively routearound, and circumvent household obstacles or radio dead spots thatmight occur. A message from node A to node C can be successfullydelivered even if the two nodes are not within range, providing that athird node B can communicate with nodes A and C. If the preferred routeis unavailable, the message originator will attempt other routes until apath is found to the “C” node. Therefore, a Z-Wave network can span muchfarther than the radio range of a single unit; however, with several ofthese hops, a delay may be introduced between the control command andthe desired result. In order for Z-Wave units to be able to routeunsolicited messages, they cannot be in sleep mode. Therefore, mostbattery-operated devices are not designed as repeater units. A Z-Wavenetwork can consist of up to 232 devices with the option of bridgingnetworks if more devices are required.

WWAN. Any wireless network herein may be a Wireless Wide Area Network(WWAN) such as a wireless broadband network, and the WWAN port may be anantenna and the WWAN transceiver may be a wireless modem. The wirelessnetwork may be a satellite network, the antenna may be a satelliteantenna, and the wireless modem may be a satellite modem. The wirelessnetwork may be a WiMAX network such as according to, compatible with, orbased on, IEEE 802.16-2009, the antenna may be a WiMAX antenna, and thewireless modem may be a WiMAX modem. The wireless network may be acellular telephone network, the antenna may be a cellular antenna, andthe wireless modem may be a cellular modem. The cellular telephonenetwork may be a Third Generation (3G) network, and may use UMTS W-CDMA,UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSMEDGE-Evolution. The cellular telephone network may be a FourthGeneration (4G) network and may use or be compatible with HSPA+, MobileWiMAX, LTE, LTE-Advanced, MBWA, or may be compatible with, or based on,IEEE 802.20-2008.

WLAN. Wireless Local Area Network (WLAN), is a popular wirelesstechnology that makes use of the Industrial, Scientific and Medical(ISM) frequency spectrum. In the US, three of the bands within the ISMspectrum are the A band, 902-928 MHz; the B band, 2.4-2.484 GHz (a.k.a.2.4 GHz); and the C band, 5.725-5.875 GHz (a.k.a. 5 GHz). Overlappingand/or similar bands are used in different regions such as Europe andJapan. In order to allow interoperability between equipment manufacturedby different vendors, few WLAN standards have evolved, as part of theIEEE 802.11 standard group, branded as WiFi (www.wi-fi.org). IEEE802.11b describes a communication using the 2.4 GHz frequency band andsupporting communication rate of 11 Mb/s, IEEE 802.11a uses the 5 GHzfrequency band to carry 54 MB/s and IEEE 802.11g uses the 2.4 GHz bandto support 54 Mb/s. The WiFi technology is further described in apublication entitled: “WiFi Technology” by Telecom Regulatory Authority,published on July 2003, which is incorporated in its entirety for allpurposes as if fully set forth herein. The IEEE 802 defines an ad-hocconnection between two or more devices without using a wireless accesspoint: the devices communicate directly when in range. An ad hoc networkoffers peer-to-peer layout and is commonly used in situations such as aquick data exchange or a multiplayer LAN game, because the setup is easyand an access point is not required.

A node/client with a WLAN interface is commonly referred to as STA(Wireless Station/Wireless client). The STA functionality may beembedded as part of the data unit, or alternatively be a dedicated unit,referred to as bridge, coupled to the data unit. While STAs maycommunicate without any additional hardware (ad-hoc mode), such networkusually involves Wireless Access Point (a.k.a. WAP or AP) as a mediationdevice. The WAP implements the Basic Stations Set (BSS) and/or ad-hocmode based on Independent BSS (IBSS). STA, client, bridge and WAP willbe collectively referred to hereon as WLAN unit. Bandwidth allocationfor IEEE 802.11g wireless in the U.S. allows multiple communicationsessions to take place simultaneously, where eleven overlapping channelsare defined spaced 5 MHz apart, spanning from 2412 MHz as the centerfrequency for channel number 1, via channel 2 centered at 2417 MHz and2457 MHz as the center frequency for channel number 10, up to channel 11centered at 2462 MHz. Each channel bandwidth is 22 MHz, symmetrically(+/−11 MHz) located around the center frequency. In the transmissionpath, first the baseband signal (IF) is generated based on the data tobe transmitted, using 256 QAM (Quadrature Amplitude Modulation) basedOFDM (Orthogonal Frequency Division Multiplexing) modulation technique,resulting a 22 MHz (single channel wide) frequency band signal. Thesignal is then up converted to the 2.4 GHz (RF) and placed in the centerfrequency of required channel, and transmitted to the air via theantenna. Similarly, the receiving path comprises a received channel inthe RF spectrum, down converted to the baseband (IF) wherein the data isthen extracted.

In order to support multiple devices and using a permanent solution, aWireless Access Point (WAP) is typically used. A Wireless Access Point(WAP, or Access Point—AP) is a device that allows wireless devices toconnect to a wired network using Wi-Fi, or related standards. The WAPusually connects to a router (via a wired network) as a standalonedevice, but can also be an integral component of the router itself.Using Wireless Access Point (AP) allows users to add devices that accessthe network with little or no cables. A WAP normally connects directlyto a wired Ethernet connection, and the AP then provides wirelessconnections using radio frequency links for other devices to utilizethat wired connection. Most APs support the connection of multiplewireless devices to one wired connection. Wireless access typicallyinvolves special security considerations, since any device within arange of the WAP can attach to the network. The most common solution iswireless traffic encryption. Modern access points come with built-inencryption such as Wired Equivalent Privacy (WEP) and Wi-Fi ProtectedAccess (WPA), typically used with a password or a passphrase.Authentication in general, and a WAP authentication in particular, isused as the basis for authorization, which determines whether aprivilege may be granted to a particular user or process, privacy, whichkeeps information from becoming known to non-participants, andnon-repudiation, which is the inability to deny having done somethingthat was authorized to be done based on the authentication. Anauthentication in general, and a WAP authentication in particular, mayuse an authentication server that provides a network service thatapplications may use to authenticate the credentials, usually accountnames and passwords of their users. When a client submits a valid set ofcredentials, it receives a cryptographic ticket that it can subsequentlybe used to access various services. Authentication algorithms includepasswords, Kerberos, and public key encryption.

Prior art technologies for data networking may be based on singlecarrier modulation techniques, such as AM (Amplitude Modulation), FM(Frequency Modulation), and PM (Phase Modulation), as well as bitencoding techniques such as QAM (Quadrature Amplitude Modulation) andQPSK (Quadrature Phase Shift Keying). Spread spectrum technologies, toinclude both DSSS (Direct Sequence Spread Spectrum) and FHSS (FrequencyHopping Spread Spectrum) are known in the art. Spread spectrum commonlyemploys Multi-Carrier Modulation (MCM) such as OFDM (OrthogonalFrequency Division Multiplexing). OFDM and other spread spectrum arecommonly used in wireless communication systems, particularly in WLANnetworks.

Bluetooth. Bluetooth is a wireless technology standard for exchangingdata over short distances (using short-wavelength UHF radio waves in theISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, andbuilding personal area networks (PANs). It can connect several devices,overcoming problems of synchronization. A Personal Area Network (PAN)may be according to, compatible with, or based on, Bluetooth™ or IEEE802.15.1-2005 standard. A Bluetooth controlled electrical appliance isdescribed in U.S. Patent Application No. 2014/0159877 to Huang entitled:“Bluetooth Controllable Electrical Appliance”, and an electric powersupply is described in U.S. Patent Application No. 2014/0070613 to Garbet al. entitled: “Electric Power Supply and Related Methods”, which areboth incorporated in their entirety for all purposes as if fully setforth herein. Any Personal Area Network (PAN) may be according to,compatible with, or based on, Bluetooth™ or IEEE 802.15.1-2005 standard.A Bluetooth controlled electrical appliance is described in U.S. PatentApplication No. 2014/0159877 to Huang entitled: “Bluetooth ControllableElectrical Appliance”, and an electric power supply is described in U.S.Patent Application No. 2014/0070613 to Garb et al. entitled: “ElectricPower Supply and Related Methods”, which are both incorporated in theirentirety for all purposes as if fully set forth herein.

Bluetooth operates at frequencies between 2402 and 2480 MHz, or 2400 and2483.5 MHz including guard bands 2 MHz wide at the bottom end and 3.5MHz wide at the top. This is in the globally unlicensed (but notunregulated) Industrial, Scientific and Medical (ISM) 2.4 GHzshort-range radio frequency band. Bluetooth uses a radio technologycalled frequency-hopping spread spectrum. Bluetooth divides transmitteddata into packets, and transmits each packet on one of 79 designatedBluetooth channels. Each channel has a bandwidth of 1 MHz. It usuallyperforms 800 hops per second, with Adaptive Frequency-Hopping (AFH)enabled. Bluetooth low energy uses 2 MHz spacing, which accommodates 40channels. Bluetooth is a packet-based protocol with a master-slavestructure. One master may communicate with up to seven slaves in apiconet. All devices share the master's clock. Packet exchange is basedon the basic clock, defined by the master, which ticks at 312.5 μsintervals. Two clock ticks make up a slot of 625 μs, and two slots makeup a slot pair of 1250 vs. In the simple case of single-slot packets themaster transmits in even slots and receives in odd slots. The slave,conversely, receives in even slots and transmits in odd slots. Packetsmay be 1, 3 or slots long, but in all cases the master's transmissionbegins in even slots and the slave's in odd slots.

A master Bluetooth device can communicate with a maximum of sevendevices in a piconet (an ad-hoc computer network using Bluetoothtechnology), though not all devices reach this maximum. The devices canswitch roles, by agreement, and the slave can become the master (forexample, a headset initiating a connection to a phone necessarily beginsas master—as initiator of the connection—but may subsequently operate asslave). The Bluetooth Core Specification provides for the connection oftwo or more piconets to form a scatternet, in which certain devicessimultaneously play the master role in one piconet and the slave role inanother. At any given time, data can be transferred between the masterand one other device (except for the little-used broadcast mode). Themaster chooses which slave device to address; typically, it switchesrapidly from one device to another in a round-robin fashion. Since it isthe master that chooses which slave to address, whereas a slave issupposed to listen in each receive slot, being a master is a lighterburden than being a slave. Being a master of seven slaves is possible;being a slave of more than one master is difficult.

Bluetooth Low Energy. Bluetooth low energy (Bluetooth LE, BLE, marketedas Bluetooth Smart) is a wireless personal area network technologydesigned and marketed by the Bluetooth Special Interest Group (SIG)aimed at novel applications in the healthcare, fitness, beacons,security, and home entertainment industries. Compared to ClassicBluetooth, Bluetooth Smart is intended to provide considerably reducedpower consumption and cost while maintaining a similar communicationrange. Bluetooth low energy is described in a Bluetooth SIG publishedDec. 2, 2014 standard Covered Core Package version: 4.2, entitled:“Master Table of Contents & Compliance Requirements—Specification Volume0”, and in an article published 2012 in Sensors [ISSN 1424-8220] byCaries Gomez et al. [Sensors 2012, 12, 11734-11753;doi:10.3390/s120211734] entitled: “Overview and Evaluation of BluetoothLow Energy: An Emerging Low-Power Wireless Technology”, which are bothincorporated in their entirety for all purposes as if fully set forthherein.

Bluetooth Smart technology operates in the same spectrum range (the2.400 GHz-2.4835 GHz ISM band) as Classic Bluetooth technology, but usesa different set of channels. Instead of the Classic Bluetooth 79 1-MHzchannels, Bluetooth Smart has 40 2-MHz channels. Within a channel, datais transmitted using Gaussian frequency shift modulation, similar toClassic Bluetooth's Basic Rate scheme. The bit rate is 1 Mbit/s, and themaximum transmit power is 10 mW. Bluetooth Smart uses frequency hoppingto counteract narrowband interference problems. Classic Bluetooth alsouses frequency hopping but the details are different; as a result, whileboth FCC and ETSI classify Bluetooth technology as an FHSS scheme,Bluetooth Smart is classified as a system using digital modulationtechniques or a direct-sequence spread spectrum. All Bluetooth Smartdevices use the Generic Attribute Profile (GATT). The applicationprogramming interface offered by a Bluetooth Smart aware operatingsystem will typically be based around GATT concepts.

Cellular. Cellular telephone network may be according to, compatiblewith, or may be based on, a Third Generation (3G) network that usesUniversal Mobile Telecommunications System (UMTS), Wideband CodeDivision Multiple Access (W-CDMA) UMTS, High Speed Packet Access (HSPA),UMTS Time-Division Duplexing (TDD), CDMA2000 1×RTT, Evolution-DataOptimized (EV-DO), Global System for Mobile communications (GSM), orEnhanced Data rates for GSM Evolution (EDGE) EDGE-Evolution. Further, acellular telephone network is a Fourth Generation (4G) network that usesEvolved High Speed Packet Access (HSPA+), Mobile WorldwideInteroperability for Microwave Access (WiMAX), Long-Term Evolution(LTE), LTE-Advanced, Mobile Broadband Wireless Access (MBWA), or isbased on IEEE 802.20-2008.

5G. 5G refers to the fifth generation technology standard for cellularnetworks, as the successor to the 4G networks which provide connectivityto most current cellphones. 5G networks are cellular networks, in whichthe service area is divided into small geographical areas called cells.All 5G wireless devices in a cell are connected to the Internet andtelephone network by radio waves through a local antenna in the cell.The main advantage of the new networks is that they will have greaterbandwidth, giving higher download speeds, eventually up to 10 gigabitsper second (Gbit/s).

The increased speed is achieved partly by using higher-frequency radiowaves than current cellular networks. However, higher-frequency radiowaves have a shorter range than the frequencies used by previous cellphone towers, requiring smaller cells. So to ensure wide service, 5Gnetworks operate on up to three frequency bands, low, medium, and high.A 5G network will be composed of networks of up to 3 different types ofcells, each requiring different antennas, each type giving a differenttradeoff of download speed vs. distance and service area. 5G cellphonesand wireless devices will connect to the network through the highestspeed antenna within range at their location:

Low-band 5G uses a similar frequency range to current 4G cellphones,600-700 MHz, giving download speeds a little higher than 4G: 30-250Megabits per Second (Mbit/s). Low-band cell towers will have a range andcoverage area similar to current 4G towers. Mid-band 5G uses microwavesof 2.5-3.7 GHz, currently allowing speeds of 100-900 Mbit/s, with eachcell tower providing service up to several miles in radius. High-band 5Gcurrently uses frequencies of 25-39 GHz, near the bottom of themillimeter wave band, although higher frequencies may be used in thefuture. It often achieves download speeds of a gigabit per second(Gbit/s), comparable to cable internet. The industry consortium settingstandards for 5G is the 3rd Generation Partnership Project (3GPP).

Random. Randomness is commonly implemented by using random numbers,defined as a sequence of numbers or symbols that lack any pattern andthus appear random, are often generated by a random number generator.Randomness for security is also described in IETF RFC 1750 “RandomnessRecommendations for Security” (December 1994), which is incorporated inits entirety for all purposes as if fully set forth herein. A randomnumber generator (having either analog or digital output) can behardware based, using a physical process such as thermal noise, shotnoise, nuclear decaying radiation, photoelectric effect or other quantumphenomena. Alternatively, or in addition, the generation of the randomnumbers can be software based, using a processor executing an algorithmfor generating pseudo-random numbers which approximates the propertiesof random numbers.

The term ‘random’ herein is intended to cover not only pure random,non-deterministically and non-predicted generated signals, but alsopseudo-random, deterministic signals such as the output of ashift-register arrangement provided with a feedback circuit as used togenerate pseudo-random binary signals or as scramblers, and chaoticsignals, and where a randomness factor may be used.

A digital random signal generator (known as random number generator)wherein numbers in binary form replaces the analog voltage value outputmay be used for any randomness. One approach to random number generationis based on using linear feedback shift registers. An example of randomnumber generators is disclosed in U.S. Pat. No. 7,124,157 to Ikakeentitled: “Random Number Generator”, in U.S. Pat. No. 4,905,176 toSchulz entitled: “Random Number Generator Circuit”, in U.S. Pat. No.4,853,884 to Brown et al. entitled: “Random Number Generator withDigital Feedback” and in U.S. Pat. No. 7,145,933 to Szajnowski entitled:“Method and Apparatus for generating Random signals”, which areincorporated in its entirety for all purposes as if fully set forthherein.

A digital random signal generator may be based on ‘True Random NumberGeneration IC RPG100/RPG100B’ available from FDK Corporation anddescribed in the data sheet ‘Physical Random number generatorRPG100.RPG100B’ REV. 08 publication number HM-RAE106-0812, which isincorporated in its entirety for all purposes as if fully set forthherein. The digital random signal generator can be hardware based,generating random numbers from a natural physical process or phenomenon,such as the thermal noise of semiconductor which has no periodicity.Typically, such hardware random number generators are based onmicroscopic phenomena such as thermal noise, shot noise, nucleardecaying radiation, photoelectric effect or other quantum phenomena, andtypically contain a transducer to convert some aspect of the physicalphenomenon to an electrical signal, an amplifier and other electronic tobring the output into a signal that can be converted into a digitalrepresentation by an analog to digital converter. In the case wheredigitized serial random number signals are generated, the output isconverted to parallel, such as 8 bits data, with 256 values of randomnumbers (values from 0 to 255). Alternatively, a digital random signalgenerator may be software (or firmware) based, such as pseudo-randomnumber generators. Such generators include a processor for executingsoftware that includes an algorithm for generating numbers, whichapproximates the properties of random numbers. The random signalgenerator (either analog or digital) may output a signal having uniformdistribution, in which there is a substantially or purely equalprobability of a signal falling between two defined limits, having noappearance outside these limits. However, Gaussian and otherdistribution may be equally used.

Microphone. A transducer is a device for converting one form of energyinto another. In an electroacoustic context, this means converting soundenergy into electrical energy (or vice versa). Electroacoustictransducers include loudspeakers, microphones, hydrophones, and sonarprojectors. These devices convert a sound pressure wave to or from anelectric signal, and the most widely used transduction principles areelectromagnetism, electrostatics and piezoelectricity. The transducersin most common loudspeakers (e.g. woofers and tweeters), areelectromagnetic devices that generate waves using a suspended diaphragmdriven by an electromagnetic voice coil, sending off pressure waves.Electret microphones and condenser microphones employ electrostatics—asthe sound wave strikes the microphone's diaphragm, it moves and inducesa voltage change. The ultrasonic systems used in medical ultrasonographyemploy piezoelectric transducers. These are made from special ceramicsin which mechanical vibrations and electrical fields are interlinkedthrough a property of the material itself.

In a common technique of acoustic measurement, acoustic signals aresampled in time, and then presented in more meaningful forms such asoctave bands or time frequency plots. The entire spectrum can be dividedinto three sections: audio, ultrasonic, and infrasonic. The audio rangefalls between 20 Hz and 20,000 Hz, and is important because itsfrequencies can be detected by the human ear. This range has a number ofapplications, including speech communication and music. The ultrasonicrange refers to the very high frequencies: 20,000 Hz and higher, andthis range has shorter wavelengths which allow better resolution inimaging technologies. On the other end of the spectrum, the lowestfrequencies are known as the infrasonic range, and these frequencies canbe used to study geological phenomena such as earthquakes.

A microphone is an electroacoustic sensor that responds to sound waves(which are essentially vibrations transmitted through an elastic solidor a liquid or gas), and converts sound into electrical energy, usuallyby means of a ribbon or diaphragm set into motion by the sound waves.The sound may be audio or audible, having frequencies in the approximaterange of 20 to 20,000 hertz, capable of being detected by human organsof hearing. Alternatively or in addition, the microphone may be used tosense inaudible frequencies, such as ultrasonic (a.k.a. ultrasound)acoustic frequencies that are above the range audible to the human ear,or above approximately 20,000 Hz. A microphone may be a condensermicrophone (a.k.a. capacitor or electrostatic microphone) where thediaphragm acts as one plate of a two plates capacitor, and thevibrations changes the distance between plates, hence changing thecapacitance. An electret microphone is a capacitor microphone based on apermanent charge of an electret or a polarized ferroelectric material. Adynamic microphone is based on electromagnetic induction, using adiaphragm attached to a small movable induction coil that is positionedin a magnetic field of a permanent magnet. The incident sound wavescause the diaphragm to vibrate, and the coil to move in the magneticfield, producing a current. Similarly, a ribbon microphone uses a thin,usually corrugated metal ribbon suspended in a magnetic field, and itsvibration within the magnetic field generates the electrical signal. Aloudspeaker is commonly constructed similar to a dynamic microphone, andthus may be used as a microphone as well. In a carbon microphone, thediaphragm vibrations apply varying pressure to a carbon, thus changingits electrical resistance. A piezoelectric microphone (a.k.a. crystal orpiezo microphone) is based on the phenomenon of piezoelectricity inpiezoelectric crystals such as potassium sodium tartrate. A microphonemay be omnidirectional, unidirectional, bidirectional, or provide otherdirectionality or polar patterns.

Noise-cancelling microphone. A noise-canceling microphone is amicrophone that is designed to filter ambient noise from the desiredsound, which is especially useful in noisy environments. The developmentis a special case of the differential microphone topology most commonlyused to achieve directionality, and all such microphones have at leasttwo ports through which sound enters; a front port normally orientedtoward the desired sound and another port that's more distant. Themicrophone's diaphragm is placed between the two ports; sound arrivingfrom an ambient sound field reaches both ports more or less equally.Sound that's much closer to the front port than to the rear will makemore of a pressure gradient between the front and back of the diaphragm,causing it to move more. The microphone's proximity effect is adjustedso that flat frequency response is achieved for sound sources very closeto the front of the mic—typically 1 to 3 cm. Sounds arriving from otherangles are subject to steep midrange and bass rolloff.

Another technique uses two or more microphones and active or passivecircuitry to reduce the noise. The primary microphone is closer to thedesired source (like a person's mouth), while a second microphonereceives ambient noise. In a noisy environment, both microphones receivenoise at a similar level, but the primary mic receives the desiredsounds more strongly. Thus if one signal is subtracted from the other(in the simplest sense, by connecting the microphones out of phase) muchof the noise is canceled while the desired sound is retained. Othertechniques may be used as well, such as using a directional primary mic,to maximize the difference between the two signals and make thecancellation easier to do. The internal electronic circuitry of anactive noise-canceling mic attempts to subtract noise signal from theprimary microphone. The circuit may employ passive or active noisecanceling techniques to filter out the noise, producing an output signalthat has a lower noise floor and a higher signal-to-noise ratio.

An improved noise canceling microphone including robust design featuresand advanced noise control and speech discrimination convergencecharacteristics is described in U.S. Pat. No. 7,248,708 to Vaudrey etal. entitled: “Noise canceling microphone”, which is incorporated in itsentirety for all purposes as if fully set forth herein. Two adaptivecontrollers are used to ensure robust performance in quickly changingacoustic environments ensuring an acceptable minimum performancecharacteristic. Additionally, a new real-time spectral estimationprocedure is applied to a noise canceling communications microphoneplatform that permits continued and optimal adaptation of non-voicebandwidth frequencies during speech transients.

Certain embodiments of a noise-cancelling microphone with acousticallytuned ports are disclosed in U.S. Pat. No. 7,162,041 to Haapapuro et al.entitled: “Noise canceling microphone with acoustically tuned ports”,which is incorporated in its entirety for all purposes as if fully setforth herein. The noise canceling microphone may comprise a housing, atransducer for converting received energy received into electricalsignals, where the transducer is located in the housing, a front andrear sound pathways to a front and rear sound openings in thetransducer, where the front and rear sound pathways may be located onopposite sides of the housing and may be displaced 180 degrees off avertical axis. The noise canceling microphone may further comprising aboom for supporting the noise canceling microphone, where the boom maybe deformed to place the noise canceling microphone near the mouth ofthe user. For example, the boom may be deformed to place the noisecanceling microphone at least ten millimeters away from the edge of themouth of the user.

Optical-based microphone. A fiber optic microphone converts acousticwaves into electrical signals by sensing changes in light intensity,instead of sensing changes in capacitance or magnetic fields as withconventional microphones. During operation, light from a laser sourcetravels through an optical fiber to illuminate the surface of areflective diaphragm. Sound vibrations of the diaphragm modulate theintensity of light reflecting off the diaphragm in a specific direction.The modulated light is then transmitted over a second optical fiber to aphoto detector, which transforms the intensity-modulated light intoanalog or digital audio for transmission or recording. Fiber opticmicrophones possess high dynamic and frequency range, similar to thebest high-fidelity conventional microphones. Fiber optic microphones donot react to or influence any electrical, magnetic, electrostatic orradioactive fields (this is called EMI/RFI immunity). The fiber opticmicrophone design is therefore ideal for use in areas where conventionalmicrophones are ineffective or dangerous, such as inside industrialturbines or in Magnetic Resonance Imaging (MRI) equipment environments.

Fiber optic microphones are robust, resistant to environmental changesin heat and moisture, and can be produced for any directionality orimpedance matching. The distance between the microphone's light sourceand its photo detector may be up to several kilometers without need forany preamplifier or other electrical device, making fiber opticmicrophones suitable for industrial and surveillance acousticmonitoring. Fiber optic microphones are used in very specificapplication areas such as for infrasound monitoring and noise-canceling.They have proven especially useful in medical applications, such asallowing radiologists, staff and patients within the powerful and noisymagnetic field to converse normally, inside the MRI suites as well as inremote control rooms. Other uses include industrial equipment monitoringand audio calibration and measurement, high-fidelity recording and lawenforcement. An example of an optical microphone is IAS MO 2000 Setavailable from Sennheiser Electronic Corporation (Headquartered in OldLyme, Conn. U.S.A.) and described in a Product Description entitled: “MO20000 Set and IAS MO 2000 Set”, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

A head for an optical microphone/sensor is disclosed in U.S. Pat. No.6,694,031 to Paritsky et al. entitled: “Optical Microphone/Sensors”,which is incorporated in its entirety for all purposes as if fully setforth herein. The head including first and second light guides; thefirst light guide being coupled at an input end to a source of light andhaving an output end portion for transmitting light onto a membrane; thesecond light guide having an input end portion for receiving lightreflected from the membrane and an output end coupled to aphotodetector; the output end and input end portions each having anupper face and side surfaces and being disposed in close proximity toeach other and optically separated along adjacent surfaces;characterized in that in order to utilize maximum light energytransmitted through the light guides by the light source, reflected bythe membrane and received by the photodetector, at least one of thefaces or surfaces is configured to extend along one or more planes whichdiffer from the plane including the axes of the transmission of thelight energy emitted from the light source and received by thephotodetector.

A method of making optical transducers A head for an opticalmicrophone/sensor is disclosed in U.S. Pat. No. 6,822,750 to Paritsky etal. entitled: “Optical Transducers and Methods of Making Same”, which isincorporated in its entirety for all purposes as if fully set forthherein. The method involves producing an integrated structure including,in a rectangular matrix array, a plurality of discrete light sources, aplurality of discrete light detectors each laterally spaced from a lightsource, a light shield in the space between a light source and a lightdetector for shielding the light detector from direct exposure to thelight source, and a transparent plastic potting material embedding thelight sources, light detectors and light shield; and cutting theintegrated structure, along lines of the matrix, into individual opticalunits, each including a light source, a light detector, a light shieldtherebetween all embedded in the transparent plastic potting material,and an optical window for outputting light from the light source and fortransmitting to the light detector light reflected back from the lightsource. Also described are optical units of a structure facilitatingmass production of such optical transducers and providing a sturdyconstruction permitting rough handling.

A small optical microphone/sensor for measuring distances to, and/orphysical properties of, a reflective surface is disclosed in U.S. Pat.No. 6,462,808 to Paritsky et al. entitled: “Small OpticalMicrophone/Sensor”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The small opticalmicrophone/sensor comprising a source of light coupled to a lightwaveguide for transmitting a light beam through the waveguide; thewaveguide having at one of its ends a pointed face with an angledetermined by Snell's Law of Refraction wherein al is the angle oftravel of the light beam through the waveguide media; α2 is the angle oftravel of the light beam in a second media when exiting from the pointedface, and n1 and n2 are the light indices of the light waveguide mediaand the second media; the reflective surface being disposed at anoptimal distance from the pointed face as determined by the angle α2;the waveguide having, at its outer surface, at least adjacent to thepointed face, means for preventing light waves impinging on the surfacefrom being reflected back into the waveguide, and a light detectorarranged to receive the light reflected from the surface.

An optical microphone for detecting an acoustic wave propagating in anambient fluid is disclosed in U.S. Pat. No. 8,813,571 to Iwamoto et al.entitled: “Optical Microphone”, which is incorporated in its entiretyfor all purposes as if fully set forth herein. The optical microphoneincluding: a propagation medium section; a light source for emitting alight wave to be transmitted through a diffraction region in thepropagation medium section; and a photoelectric conversion section fordetecting the light wave having been transmitted through the propagationmedium section. A first acoustic wave which is a portion of the acousticwave and a second acoustic wave which is another portion thereof areallowed to propagate in the propagation medium section so as tosimultaneously arrive at the diffraction region, and an interferencecomponent between a +1st order diffracted light wave and a −1st orderdiffracted light wave of the light wave generated based on a refractiveindex distribution of the propagation medium occurring in thediffraction region.

Sensors array. Multiple sensors may be used arranged as a sensor array(such as linear sensor array), for improving the sensitivity, accuracy,resolution, and other parameters of the sensed phenomenon. The sensorarray may be directional, and better measure the parameters of theimpinging signal to the array, such as the number, magnitudes,frequencies, Direction-Of-Arrival (DOA), distances, and speeds of thesignals. The processing of the entire sensor array outputs, such as toobtain a single measurement or a single parameter, may be performed by adedicated processor, which may be part of the sensor array assembly. Thesame component may serve both as a sensor and as actuator, such asduring different times, and may be associated with the same or differentphenomenon. A sensor operation may be based on an external or integralmechanism for generating a stimulus or an excitation to generateinfluence or create a phenomenon.

Microphone array. A microphone array is any number of microphonesoperating in tandem. There are many applications such as systems forextracting voice input from ambient noise (notably telephones, speechrecognition systems, hearing aids), surround sound and relatedtechnologies, binaural recording, locating objects by sound: acousticsource localization, e.g., military use to locate the source(s) ofartillery fire, aircraft location, and tracking, and high fidelityoriginal recordings. Typically, an array is made up of omnidirectionalmicrophones, directional microphones, or a mix of omnidirectional anddirectional microphones distributed about the perimeter of a space,linked to a computer that records and interprets the results into acoherent form. Arrays may also be formed using numbers of very closelyspaced microphones. Given a fixed physical relationship in space betweenthe different individual microphone transducer array elements,simultaneous DSP (digital signal processor) processing of the signalsfrom each of the individual microphone array elements can create one ormore “virtual” microphones. Different algorithms permit the creation ofvirtual microphones with extremely complex virtual polar patterns andeven the possibility to steer the individual lobes of the virtualmicrophones patterns to home-in-on, or to reject, particular sources ofsound.

In case the array consists of omnidirectional microphones they acceptsound from all directions, so electrical signals of the microphonescontain the information about the sounds coming from all directions.Joint processing of these sounds allow selecting the sound signal comingfrom the given direction. Hence, microphone array selects the soundcoming from a given direction by processing multichannel signals. Usingmicrophone arrays is described in an article by Rainer Zelinski (of theDeutsche Bundespost, Research Institute Berlin) published 1998 by IEEE(CH2561-9/88/0000-2578) entitled: “A Microphone Array with AdaptivePost-Filtering for Noise Reduction in Reverberant Rooms”, and in apresentation by Sven Fischer et al. presented Dec. 2-6, 1996 at the 3rdJoint Meeting of the Acoustical Society of America and the AcousticalSociety of Japan entitled: “Adaptive Microphone Arrays for SpeechEnhancement in Coherent and Incoherent Noise Fields”, which are bothincorporated in their entirety for all purposes as if fully set forthherein.

Display. A display is used for presentation of visual data orinformation, commonly on a screen. A display is typically consists of anarray of light emitters (typically in a matrix form), and commonlyprovides a visual depiction of a single, integrated, or organized set ofinformation, such as text, graphics, image or video. A display may be amonochrome (a.k.a. black-and-white) type, which typically displays twocolors, one for the background and one for the foreground. Old computermonitor displays commonly use black and white, green and black, or amberand black. A display may be a gray-scale type, which is capable ofdisplaying different shades of gray, or may be a color type, capable ofdisplaying multiple colors, anywhere from 16 to over many millionsdifferent colors, and may be based on Red, Green, and Blue (RGB)separate signals. A video display is designed for presenting videocontent. The screen is the actual location where the information isactually optically visualized by humans. The screen may be an integralpart of the display.

Alternatively or in addition, the display may be an image or videoprojector, that projects an image (or a video consisting of movingimages) onto a screen surface, which is a separate component and is notmechanically enclosed with the display housing. Most projectors createan image by shining a light through a small transparent image, but somenewer types of projectors can project the image directly, by usinglasers. A projector may be based on an Eidophor, Liquid Crystal onSilicon (LCoS or LCOS), or LCD, or may use Digital Light Processing(DLP™) technology, and may further be MEMS based. A virtual retinaldisplay, or retinal projector, is a projector that projects an imagedirectly on the retina instead of using an external projection screen.

Common display resolutions used today include SVGA (800×600 pixels), XGA(1024×768 pixels), 720p (1280×720 pixels), and 1080p (1920×1080 pixels).Standard-Definition (SD) standards, such as used in SD Television(SDTV), are referred to as 576i, derived from the European-developed PALand SECAM systems with 576 interlaced lines of resolution; and 480i,based on the American National Television System Committee (ANTSC) NTSCsystem. High-Definition (HD) video refers to any video system of higherresolution than standard-definition (SD) video, and most commonlyinvolves display resolutions of 1,280×720 pixels (720p) or 1,920×1,080pixels (1080i/1080p). A display may be a 3D (3-Dimensions) display,which is the display device capable of conveying a stereoscopicperception of 3-D depth to the viewer. The basic technique is to presentoffset images that are displayed separately to the left and right eye.Both of these 2-D offset images are then combined in the brain to givethe perception of 3-D depth. The display may present the information asscrolling, static, bold or flashing.

A display may be an analog display having an analog signal input. Analogdisplays are commonly using interfaces such as composite video such asNTSC, PAL or SECAM formats. Similarly, analog RGB, VGA (Video GraphicsArray), SVGA (Super Video Graphics Array), SCART, S-video and otherstandard analog interfaces can be used. Alternatively or in addition, adisplay may be a digital display, having a digital input interface.Standard digital interfaces such as an IEEE1394 interface (a.k.a.FireWire™), may be used. Other digital interfaces that can be used areUSB, SDI (Serial Digital Interface), HDMI (High-Definition MultimediaInterface), DVI (Digital Visual Interface), UDI (Unified DisplayInterface), DisplayPort, Digital Component Video and DVB (Digital VideoBroadcast). In some cases, an adaptor is required in order to connect ananalog display to the digital data. For example, the adaptor may convertbetween composite video (PAL, NTSC) or S-Video and DVI or HDTV signal.Various user controls can be available to allow the user to control andeffect the display operations, such as an on/off switch, a reset buttonand others. Other exemplary controls involve display associated settingssuch as contrast, brightness and zoom.

A display may be a Cathode-Ray Tube (CRT) display, which is based onmoving an electron beam back and forth across the back of the screen.Such a display commonly comprises a vacuum tube containing an electrongun (a source of electrons), and a fluorescent screen used to viewimages. It further has a means to accelerate and deflect the electronbeam onto the fluorescent screen to create the images. Each time thebeam makes a pass across the screen, it lights up phosphor dots on theinside of the glass tube, thereby illuminating the active portions ofthe screen. By drawing many such lines from the top to the bottom of thescreen, it creates an entire image. A CRT display may be a shadow maskor an aperture grille type.

A display may be a Liquid Crystal Display (LCD) display, which utilizetwo sheets of polarizing material with a liquid crystal solution betweenthem. An electric current passed through the liquid causes the crystalsto align so that light cannot pass through them. Each crystal,therefore, is like a shutter, either allowing a backlit light to passthrough or blocking the light. In monochrome LCD, images usually appearas blue or dark gray images on top of a grayish-white background. ColorLCD displays commonly use passive matrix and Thin Film Transistor (TFT)(or active-matrix) for producing color. Recent passive-matrix displaysare using new CSTN and DSTN technologies to produce sharp colorsrivaling active-matrix displays.

Some LCD displays use Cold-Cathode Fluorescent Lamps (CCFLs) forbacklight illumination. An LED-backlit LCD is a flat panel display thatuses LED backlighting instead of the cold cathode fluorescent (CCFL)backlighting, allowing for a thinner panel, lower power consumption,better heat dissipation, a brighter display, and better contrast levels.Three forms of LED may be used: White edge-LEDs around the rim of thescreen, using a special diffusion panel to spread the light evenlybehind the screen (the most usual form currently), an array of LEDsarranged behind the screen whose brightness are not controlledindividually, and a dynamic “local dimming” array of LEDs that arecontrolled individually or in clusters to achieve a modulated backlightlight pattern. A Blue Phase Mode LCD is an LCD technology that useshighly twisted cholesteric phases in a blue phase, in order to improvethe temporal response of liquid crystal displays (LCDs).

A Field Emission Display (FED) is a display technology that useslarge-area field electron emission sources to provide the electrons thatstrike colored phosphor, to produce a color image as an electronicvisual display. In a general sense, a FED consists of a matrix ofcathode ray tubes, each tube producing a single sub-pixel, grouped inthrees to form red-green-blue (RGB) pixels. FEDs combine the advantagesof CRTs, namely their high contrast levels and very fast response times,with the packaging advantages of LCD and other flat panel technologies.They also offer the possibility of requiring less power, about half thatof an LCD system. FED display operates like a conventional cathode raytube (CRT) with an electron gun that uses high voltage (10 kV) toaccelerate electrons which in turn excite the phosphors, but instead ofa single electron gun, a FED display contains a grid of individualnanoscopic electron guns. A FED screen is constructed by laying down aseries of metal stripes onto a glass plate to form a series of cathodelines.

A display may be an Organic Light-Emitting Diode (OLED) display, adisplay device that sandwiches carbon-based films between two chargedelectrodes, one a metallic cathode and one a transparent anode, usuallybeing glass. The organic films consist of a hole-injection layer, ahole-transport layer, an emissive layer and an electron-transport layer.When voltage is applied to the OLED cell, the injected positive andnegative charges recombine in the emissive layer and create electroluminescent light. Unlike LCDs, which require backlighting, OLEDdisplays are emissive devices—they emit light rather than modulatetransmitted or reflected light. There are two main families of OLEDs:those based on small molecules and those employing polymers. Addingmobile ions to an OLED creates a light-emitting electrochemical cell orLEC, which has a slightly different mode of operation. OLED displays canuse either Passive-Matrix (PMOLED) or active-matrix addressing schemes.Active-Matrix OLEDs (AMOLED) require a thin-film transistor backplane toswitch each individual pixel on or off, but allow for higher resolutionand larger display sizes.

A display may be an Electroluminescent Displays (ELDs) type, which is aflat panel display created by sandwiching a layer of electroluminescentmaterial such as GaAs between two layers of conductors. When currentflows, the layer of material emits radiation in the form of visiblelight. Electroluminescence (EL) is an optical and electrical phenomenonwhere a material emits light in response to an electric current passedthrough it, or to a strong electric field.

A display may be based on an Electronic Paper Display (EPD) (a.k.a.e-paper and electronic ink) display technology which is designed tomimic the appearance of ordinary ink on paper. Unlike conventionalbacklit flat panel displays which emit light, electronic paper displaysreflect light like ordinary paper. Many of the technologies can holdstatic text and images indefinitely without using electricity, whileallowing images to be changed later. Flexible electronic paper usesplastic substrates and plastic electronics for the display backplane.

An EPD may be based on Gyricon technology, using polyethylene spheresbetween 75 and 106 micrometers across. Each sphere is a janus particlecomposed of negatively charged black plastic on one side and positivelycharged white plastic on the other (each bead is thus a dipole). Thespheres are embedded in a transparent silicone sheet, with each spheresuspended in a bubble of oil so that they can rotate freely. Thepolarity of the voltage applied to each pair of electrodes thendetermines whether the white or black side is face-up, thus giving thepixel a white or black appearance. Alternatively or in addition, an EPDmay be based on an electrophoretic display, where titanium dioxide(Titania) particles approximately one micrometer in diameter aredispersed in hydrocarbon oil. A dark-colored dye is also added to theoil, along with surfactants and charging agents that cause the particlesto take on an electric charge. This mixture is placed between twoparallel, conductive plates separated by a gap of 10 to 100 micrometers.When a voltage is applied across the two plates, the particles willmigrate electrophoretically to the plate bearing the opposite chargefrom that on the particles.

Further, an EPD may be based on Electro-Wetting Display (EWD), which isbased on controlling the shape of a confined water/oil interface by anapplied voltage. With no voltage applied, the (colored) oil forms a flatfilm between the water and a hydrophobic (water-repellent) insulatingcoating of an electrode, resulting in a colored pixel. When a voltage isapplied between the electrode and the water, it changes the interfacialtension between the water and the coating. As a result, the stackedstate is no longer stable, causing the water to move the oil aside.Electrofluidic displays are a variation of an electrowetting display,involving the placing of aqueous pigment dispersion inside a tinyreservoir. Voltage is used to electromechanically pull the pigment outof the reservoir and spread it as a film directly behind the viewingsubstrate. As a result, the display takes on color and brightnesssimilar to that of conventional pigments printed on paper. When voltageis removed liquid surface tension causes the pigment dispersion torapidly recoil into the reservoir.

A display may be a Vacuum Fluorescent Display (VFD) that emits a verybright light with high contrast and can support display elements ofvarious colors. VFDs can display seven-segment numerals, multi-segmentalphanumeric characters or can be made in a dot-matrix to displaydifferent alphanumeric characters and symbols.

A display may be a laser video display or a laser video projector. ALaser display requires lasers in three distinct wavelengths—red, green,and blue. Frequency doubling can be used to provide the greenwavelengths, and a small semiconductor laser such asVertical-External-Cavity Surface-Emitting-Laser (VECSEL) or aVertical-Cavity Surface-Emitting Laser (VCSEL) may be used. Severaltypes of lasers can be used as the frequency doubled sources: fiberlasers, inter cavity doubled lasers, external cavity doubled lasers,eVCSELs, and OPSLs (Optically Pumped Semiconductor Lasers). Among theinter-cavity doubled lasers VCSELs have shown much promise and potentialto be the basis for a mass-produced frequency doubled laser. A VECSEL isa vertical cavity, and is composed of two mirrors. On top of one of themis a diode as the active medium. These lasers combine high overallefficiency with good beam quality. The light from the high-powerIR-laser diodes is converted into visible light by means of extra-cavitywaveguided second harmonic generation. Laser-pulses with about 10 KHzrepetition rate and various lengths are sent to a Digital MicromirrorDevice where each mirror directs the pulse either onto the screen orinto the dump.

A display may be a segment display, such as a numerical or analphanumerical display that can show only digits or alphanumericcharacters, commonly composed of several segments that switch on and offto give the appearance of desired glyph, The segments are usually singleLEDs or liquid crystals, and may further display visual display materialbeyond words and characters, such as arrows, symbols, ASCII andnon-ASCII characters. Non-limiting examples are Seven-segment display(digits only), Fourteen-segment display, and Sixteen-segment display. Adisplay may be a dot matrix display, used to display information onmachines, clocks, railway departure indicators and many other devicesrequiring a simple display device of limited resolution. The displayconsists of a matrix of lights or mechanical indicators arranged in arectangular configuration (other shapes are also possible, although notcommon) such that by switching on or off selected lights, text orgraphics can be displayed. A dot matrix controller converts instructionsfrom a processor into signals which turns on or off the lights in thematrix so that the required display is produced.

Sounder. A sounder converts electrical energy to sound waves transmittedthrough the air, an elastic solid material, or a liquid, usually bymeans of a vibrating or moving ribbon or diaphragm. The sound may beaudio or audible, having frequencies in the approximate range of 20 to20,000 hertz, capable of being detected by human organs of hearing.Alternatively or in addition, the sounder may be used to emit inaudiblefrequencies, such as ultrasonic (a.k.a. ultrasound) acoustic frequenciesthat are above the range audible to the human ear, or aboveapproximately 20,000 Hz. A sounder may be omnidirectional,unidirectional, bidirectional, or provide other directionality or polarpatterns.

A loudspeaker (a.k.a. speaker) is a sounder that produces sound inresponse to an electrical audio signal input, typically audible sound.The most common form of loudspeaker is the electromagnetic (or dynamic)type, uses a paper cone supporting a moving voice coil electromagnetacting on a permanent magnet. Where accurate reproduction of sound isrequired, multiple loudspeakers may be used, each reproducing a part ofthe audible frequency range. A loudspeaker is commonly optimized formiddle frequencies; tweeters for high frequencies; and sometimessupertweeter is used which is optimized for the highest audiblefrequencies.

A loudspeaker may be a piezo (or piezoelectric) speaker contains apiezoelectric crystal coupled to a mechanical diaphragm and is based onthe piezoelectric effect. An audio signal is applied to the crystal,which responds by flexing in proportion to the voltage applied acrossthe crystal surfaces, thus converting electrical energy into mechanical.Piezoelectric speakers are frequently used as beepers in watches andother electronic devices, and are sometimes used as tweeters inless-expensive speaker systems, such as computer speakers and portableradios. A loudspeaker may be a magnetostrictive transducers, based onmagnetostriction, have been predominantly used as sonar ultrasonic soundwave radiators, but their usage has spread also to audio speakersystems.

A loudspeaker may be an Electro-Static Loudspeaker (ESL), in which soundis generated by the force exerted on a membrane suspended in anelectrostatic field. Such speakers use a thin flat diaphragm usuallyconsisting of a plastic sheet coated with a conductive material such asgraphite sandwiched between two electrically conductive grids, with asmall air gap between the diaphragm and grids. The diaphragm is usuallymade from a polyester film (thickness 2-20 μm) with exceptionalmechanical properties, such as PET film. By means of the conductivecoating and an external high voltage supply the diaphragm is held at aDC potential of several kilovolts with respect to the grids. The gridsare driven by the audio signal; and the front and rear grids are drivenin antiphase. As a result, a uniform electrostatic field proportional tothe audio signal is produced between both grids. This causes a force tobe exerted on the charged diaphragm, and its resulting movement drivesthe air on either side of it.

A loudspeaker may be a magnetic loudspeaker, and may be a ribbon orplanar type, is based on a magnetic field. A ribbon speaker consists ofa thin metal-film ribbon suspended in a magnetic field. The electricalsignal is applied to the ribbon, which moves with it to create thesound. Planar magnetic speakers are speakers with roughly rectangularflat surfaces that radiate in a bipolar (i.e., front and back) manner,and may be having printed or embedded conductors on a flat diaphragm.Planar magnetic speakers consist of a flexible membrane with a voicecoil printed or mounted on it. The current flowing through the coilinteracts with the magnetic field of carefully placed magnets on eitherside of the diaphragm, causing the membrane to vibrate more uniformlyand without much bending or wrinkling. A loudspeaker may be a bendingwave loudspeaker, which uses a diaphragm that is intentionally flexible.

A sounder may an electromechanical type, such as an electric bell, whichmay be based on an electromagnet, causing a metal ball to clap on cup orhalf-sphere bell. A sounder may be a buzzer (or beeper), a chime, awhistle or a ringer. Buzzers may be either electromechanical orceramic-based piezoelectric sounders which make a high-pitch noise, andmay be used for alerting. The sounder may emit a single or multipletones, and can be in continuous or intermittent operation.

In one example, the sounder is used to play a stored digital audio. Thedigital audio content can be stored in the sounder, the actuator unit,the router, the control server, or any combination thereof. Further, fewfiles may be stored (e.g., representing different announcements orsongs), selected by the control logic. Alternatively or in addition, thedigital audio data may be received by the sounder, the actuator unit,the router, the control server, or any combination thereof, fromexternal sources via the above networks. Furthermore, the source of thedigital audio may a microphone serving as a sensor, either afterprocessing, storing, delaying, or any other manipulation, or asoriginally received resulting ‘doorphone’ or ‘intercom’ functionalitybetween a microphone and a sounder in the building.

In another example, the sounder simulates the voice of a human being orgenerates music, typically by using an electronic circuit having amemory for storing the sounds (e.g., music, song, voice message, etc.),a digital to analog converter to reconstruct the electricalrepresentation of the sound, and a driver for driving a loudspeaker,which is an electro-acoustic transducer that converts an electricalsignal to sound. An example of a greeting card providing music andmechanical movement is disclosed in U.S. Patent Application No.2007/0256337 to Segan entitled: “User Interactive Greeting Card”, whichis incorporated in its entirety for all purposes as if fully set forthherein.

It is noted that the expression “sound information” or “sound” as usedherein may refer to acoustic wave energy produced when playing aninstrument and/or singing. Some systems and methods may also operate instatic mode, in which no BGM music is played, and the user plays thenotes at his own pace. In some embodiments, a cursor may be displayed,which advances in accordance with the user's playing or with theexpected pace.

In one example, the system is used for sound or music generation. Forexample, the sound produced can emulate the sounds of a conventionalacoustical music instrument, such as a piano, tuba, harp, violin, flute,guitar and so forth. In one example, the sounder is an audible signalingdevice, emitting audible sounds that can be heard (having frequencycomponents in the 20-20,000 Hz band). In one example the sound generatedis music or song. The elements of the music such as pitch (which governsmelody and harmony), rhythm (and its associated concepts tempo, meter,and articulation), dynamics, and the sonic qualities of timbre andtexture, may be associated with the shape theme. For example, if amusical instrument shown in the picture, the music generated by thatinstrument will be played, e.g., drumming sound of drums and playing ofa flute or guitar. In one example, a talking human voice is played bythe sounder. The sound may be a syllable, a word, a phrase, a sentence,a short story or a long story, and can be based on speech synthesis orprerecorded. Male or female voice can be used, further being young orold.

Some examples of toys that include generation of an audio signal such asmusic are disclosed in U.S. Pat. No. 4,496,149 to Schwartzberg entitled:“Game Apparatus Utilizing Controllable Audio Signals”, in U.S. Pat. No.4,516,260 to Breedlove et al. entitled: “Electronic Learning Aid or Gamehaving Synthesized Speech”, in U.S. Pat. No. 7,414,186 to Scarpa et al.entitled: “System and Method for Teaching Musical Notes”, in U.S. Pat.No. 4,968,255 to Lee et al., entitled: “Electronic InstructionalApparatus”, in U.S. Pat. No. 4,248,123 to Bunger et al., entitled:“Electronic Piano” and in U.S. Pat. No. 4,796,891 to Milner entitled:“Musical Puzzle Using Sliding Tiles”, and toys with means forsynthesizing human voice are disclosed in U.S. Pat. No. 6,527,611 toCummings entitled: “Place and Find Toy”, and in U.S. Pat. No. 4,840,602to Rose entitled: “Talking Doll Responsive to External Signal”, whichare all incorporated in their entirety for all purposes as if fully setforth herein. A music toy kit combining music toy instrument with a setof construction toy blocks is disclosed in U.S. Pat. No. 6,132,281 toKlitsner et al. entitled: “Music Toy Kit” and in U.S. Pat. No. 5,349,129to Wisniewski et al. entitled: “Electronic Sound Generating Toy”, whichare incorporated in their entirety for all purposes as if fully setforth herein.

A speech synthesizer used to produce natural and intelligible artificialhuman speech may be implemented in hardware, in software, or combinationthereof. A speech synthesizer may be Text-To-Speech (TTS) based, thatconverts normal language text to speech, or alternatively (or inaddition) may be based on rendering symbolic linguistic representationlike phonetic transcription. A TTS typically involves two steps, thefront-end where the raw input text is pre-processed to fully write-outwords replacing numbers and abbreviations, followed by assigningphonetic transcriptions to each word (text-to-phoneme), and the back-end(or synthesizer) where the symbolic linguistic representation isconverted to output sound.

The generating of synthetic speech waveform typically uses aconcatenative or formant synthesis. The concatenative synthesis commonlyproduces the most natural-sounding synthesized speech, and is based onthe concatenation (or stringing together) of segments of recordedspeech. There are three main types of concatenative synthesis: Unitselection, diphone synthesis, and domain-specific synthesis. Unitselection synthesis is based on large databases of recorded speechincluding individual phones, diphones, half-phones, syllables,morphemes, words, phrases, and sentences, indexed based on thesegmentation and acoustic parameters like the fundamental frequency(pitch), duration, position in the syllable, and neighboring phones. Atrun time, the desired target utterance is created by determining(typically using a specially weighted decision tree) the best chain ofcandidate units from the database (unit selection). Diphone synthesisuses a minimal speech database containing all the diphones(sound-to-sound transitions) occurring in a language, and at runtime,the target prosody of a sentence is superimposed on these minimal unitsby means of digital signal processing techniques such as linearpredictive coding. Domain-specific synthesis is used where the output islimited to a particular domain, using concatenates prerecorded words andphrases to create complete utterances. In formant synthesis thesynthesized speech output is created using additive synthesis and anacoustic model (physical modeling synthesis), rather than on using humanspeech samples. Parameters such as fundamental frequency, voicing, andnoise levels are varied over time to create a waveform of artificialspeech. The synthesis may further be based on articulatory synthesiswhere computational techniques for synthesizing speech are based onmodels of the human vocal tract and the articulation processes occurringthere, or may be HMM-based synthesis which is based on hidden Markovmodels, where the frequency spectrum (vocal tract), fundamentalfrequency (vocal source), and duration (prosody) of speech are modeledsimultaneously by HMMs and generated based on the maximum likelihoodcriterion. The speech synthesizer may further be based on the bookentitled: “Development in Speech Synthesis”, by Mark Tatham andKatherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN:0-470-85538-X, and on the book entitled: “Speech Synthesis andRecognition” by John Holmes and Wendy Holmes, 2^(nd) Edition, published2001 ISBN: 0-7484-0856-8, which are both incorporated in their entiretyfor all purposes as if fully set forth herein.

Haptic. Haptic technology, also known as kinaesthetic communication or3D touch, refers to any technology that can create an experience oftouch by applying forces, vibrations, or motions to the user. Thesetechnologies can be used to create virtual objects in a computersimulation, to control virtual objects, and to enhance remote control ofmachines and devices. Haptic devices may incorporate tactile sensorsthat measure forces exerted by the user on the interface, and simplehaptic devices are common in the form of game controllers, joysticks,and steering wheels. Haptic technology facilitates investigation of howthe human sense of touch works by allowing the creation of controlledhaptic virtual objects. In general, three sensory systems related tosense of touch in humans are distinguished: cutaneous, kinaesthetic, andhaptic.

The majority of electronics offering haptic feedback use vibrations, andmost use a type of Eccentric Rotating Mass (ERM) actuator, consisting ofan unbalanced weight attached to a motor shaft. As the shaft rotates,the spinning of this irregular mass causes the actuator and the attacheddevice to shake. Some devices accomplish their vibrations with a LinearResonant Actuator (LRA), which moves a mass in a reciprocal manner bymeans of a magnetic voice coil, similar to how AC electrical signals aretranslated into motion in the cone of a loudspeaker. LRAs are capable ofquicker response times than ERMs, and thus can transmit more accuratehaptic imagery. Piezoelectric actuators are also employed to producevibrations, and offer even more precise motion than LRAs, with lessnoise and in a smaller platform, but require higher voltages than doERMs and LRAs.

Some devices use motors to manipulate the movement of an item held bythe user. A common use is in automobile driving video games andsimulators, which turn the steering wheel to simulate forces experiencedwhen cornering a real vehicle. Air vortex rings are donut-shaped airpockets made up of concentrated gusts of air. Focused air vortices canhave the force to blow out a candle or disturb papers from a few yardsaway. Focused ultrasound beams can be used to create a localized senseof pressure on a finger without touching any physical object. The focalpoint that creates the sensation of pressure is generated byindividually controlling the phase and intensity of each transducer inan array of ultrasound transducers. These beams can also be used todeliver sensations of vibration, and to give users the ability to feelvirtual 3D objects.

Haptic rendering is described in a paper entitled: “Introduction toHaptic Rendering” by Miguel A. Otaduy and Ming C. Lin, published July2005 [DOI: 10.1145/1198555.1198603], which is incorporated in itsentirety for all purposes as if fully set forth herein. The term hapticthe adjective used to describe something relating to or based on thesense of touch. Haptic is to touching as visual is to seeing and asauditory is to hearing. Typically, touch is one of the main avenues ofsensation, and it can be divided into cutaneous, kinesthetic, and hapticsystems, based on the underlying neural inputs. The cutaneous systememploys receptors embedded in the skin, while the kinesthetic systememploys receptors located in muscles, tendons, and joints. The hapticsensory system employs both cutaneous and kinesthetic receptors, but itdiffers in the sense that it is associated with an active procedure.Touch becomes active when the sensory inputs are combined withcontrolled body motion. For example, cutaneous touch becomes active whenwe explore a surface or grasp an object, while kinesthetic touch becomesactive when we manipulate an object and touch other objects with it.Haptic rendering is defined as the process of computing and generatingforces in response to user interactions with virtual objects. Severalhaptic rendering algorithms consider the paradigm of touching virtualobjects with a single contact point. Rendering algorithms that followthis description are called 3-DoF haptic rendering algorithms, because apoint in 3D has only three DoFs. Other haptic rendering algorithms dealwith the problem of rendering the forces and torques arising from theinteraction of two virtual objects. This problem is called 6-DoF hapticrendering, because the grasped object has six DoFs (position andorientation in 3D), and the haptic feedback comprises 3D force andtorque. When we eat with a fork, write with a pen, or open a lock with akey, we are moving an object in 3D, and we feel the interaction withother objects. This is, in essence, 6-DoF object manipulation withforce-and-torque feedback.

Characteristics of kinesthetic systems including their main function,historical evolution, commonly used technologies for construction andspecific cases, and an analogical development of tactile systems, aredescribed in an article entitled: “HAPTIC INTERFACES: KINESTHETIC VS.TACTILE SYSTEMS” by Zasúlich Perez Ariza and Mauricio Santís-Chavespublished 2016 [DOI: https://doi.org/10.24050/reia.v13i26.1065] in EIA,ISSN 1794-1237/Year XIII/Volume 13/Issue N.26/July-December 2016, whichis incorporated in its entirety for all purposes as if fully set forthherein.

FFT. A Fast Fourier Transform (FFT) algorithm computes the discreteFourier transform (DFT) of a sequence, or the inverse. Fourier analysisconverts a signal from its original domain (often time or space) to arepresentation in the frequency domain and vice versa. An FFT rapidlycomputes such transformations by factorizing the DFT matrix into aproduct of sparse (mostly zero) factors. The DFT is obtained bydecomposing a sequence of values into components of differentfrequencies. This operation is useful in many fields (see discreteFourier transform for properties and applications of the transform) butcomputing it directly from the definition is often too slow to bepractical. An FFT is a way to compute the same result more quickly:computing the DFT of N points in the naive way, using the definition,takes O(N2) arithmetical operations, while an FFT can compute the sameDFT in only O(N log N) operations. The difference in speed can beenormous, especially for long data sets where N may be in the thousandsor millions. In practice, the computation time can be reduced by severalorders of magnitude in such cases, and the improvement is roughlyproportional to N/log N. This huge improvement made the calculation ofthe DFT practical; FFTs are of great importance to a wide variety ofapplications, from digital signal processing and solving partialdifferential equations to algorithms for quick multiplication of largeintegers.

By far the most commonly used FFT is the Cooley-Tukey algorithm. This isa divide and conquer algorithm that recursively breaks down a DFT of anycomposite size N=N1N2 into many smaller DFTs of sizes N1 and N2, alongwith O(N) multiplications by complex roots of unity traditionally calledtwiddle factors. The best known use of the Cooley-Tukey algorithm is todivide the transform into two pieces of size N/2 at each step, and istherefore limited to power-of-two sizes, but any factorization can beused in general (as was known to both Gauss and Cooley/Tukey). These arecalled the radix-2 and mixed-radix cases, respectively (and othervariants such as the split-radix FFT have their own names as well).Although the basic idea is recursive, most traditional implementationsrearrange the algorithm to avoid explicit recursion. In addition,because the Cooley-Tukey algorithm breaks the DFT into smaller DFTs, itcan be combined arbitrarily with any other algorithm for the DFT, suchas those described below. FFT is described in an article by PaulHeckbert dated February 1995 (Revised 27 Jan. 1998) [Notes 3, ComputerGraphics 2, 15-463] entitled: “Fourier Transforms and the Fast FourierTransform (FFT) Algorithm”, and in Freescale Semiconductor, Inc.Application Note, Document Number AN4255 Rev. 4, 07/2015, entitled:“FFT-Based Algorithm for Metering Applications”, which are bothincorporated in their entirety for all purposes as if fully set forthherein.

Wearable. As used herein, the term “wearable device” (or “wearable”)includes a body-borne device (or item) designed or intended to be wornby a human. Such devices are typically comfortably worn on, and arecarried or transported by, the human body, and are commonly used tocreate constant, convenient, seamless, portable, and mostly hands-freeaccess to electronics and computers. The wearable devices may be indirect contact with the human body (such as by touching, or attachingto, the body skin), or may be releasably attachable to clothes or otheritems intended or designed to be worn on the human body. In general, thegoal of wearable technologies is to smoothly incorporate functional,portable electronics and computers into individuals' daily lives.Wearable devices may be releasably attached to the human body usingattaching means such as straps, buckles, belts, or clasps. Alternativelyor in addition, wearable devices may be shaped, structured, or having aform factor to be body releasably mountable or attachable, such as usingeye-glass frames or headphones. Further, wearable devices may be wornunder, with, or on top of, clothing.

Wearable devices may interact as sensors or actuators with an organ orpart of the human body, such as a head mounted wearable device mayinclude a screen suspended in front of a user's eye, without providingany aid to the user's vision. Examples of wearable devices includewatches, glasses, contact lenses, pedometers, chest straps, wrist-bands,head bands, arm bands, belt, head wear, hats, glasses, watches,sneakers, clothing, pads, e-textiles and smart fabrics, headbands,beanies, and caps, as well as jewelry such as rings, bracelets, andhearing aid-like devices that are designed to look like earrings. Awearable device may be structured, designed, or have a form factor thatis identical to, substantially similar to, or is at least in partsubstitute to, a traditional wearable item.

A wearable device may be a headwear that may be structured, designed, orhave a form factor that is identical to, substantially similar to, or isat least in part substitute to, any headwear item. The headwear may beattached to, or be in contact with, a head part, such as a face, nose,right nostril, left nostril, right cheek, left cheek, right eye, lefteye, right ear, or left ear, nose, mouth, lip, forehead, or chin. Awearable device may be structured, designed, or have a form factor thatis identical to, substantially similar to, or is at least in partsubstitute to, a bonnet, a cap, a crown, a fillet, a hair cover, a hat,a helmet, a hood, a mask, a turban, a veil, or a wig.

A headwear device may be an eyewear that may be structured, designed, orhave a form factor that is identical to, substantially similar to, or isat least in part substitute to, any eyewear item, such as glasses,sunglasses, a contact lens, a blindfold, or a goggle. A headwear devicemay be an earpiece that may be structured, designed, or have a formfactor that is identical to, substantially similar to, or is at least inpart substitute to, any earpiece item, such as a hearing aid, aheadphone, a headset, or an earplug.

A wearable device may be releasably or permanently attach to, or be partof, a clothing article such as a tie, sweater, jacket, or hat. Theattachment may use taping, gluing, pinning, enclosing, encapsulating, orany other method of attachment or integration known in the art.Furthermore, in some embodiments, there may be an attachment elementsuch as a pin or a latch and hook system, of portion thereof (with thecomplementary element on the item to which it is to be affixed) or clip.In a non-limiting example, the attachment element has a clip-like designto allow attachment to pockets, belts, watches, bracelets, broaches,rings, shoes, hats, bike handles, necklaces, ties, spectacles, collars,socks, bags, purses, wallets, or cords.

A wearable device may be releasably or permanently attach to, or be partof, a top underwear such as a bra, camisole, or undershirt, a bottomunderwear such as a diaper, panties, plastic pants, slip, thong,underpants, boxer briefs, boxer shorts, or briefs, or a full-bodyunderwear such as bodysuit, long underwear, playsuit, or teddy.Similarly, a wearable device may be releasably or permanently attach to,or be part of, a headwear such as a Baseball cap, Beret, Cap, Fedora,hat, helmet, hood, knit cap, toque, turban, or veil. Similarly, awearable device may be releasably or permanently attach to, or be partof, a footwear such as an athletic shoe, boot, court shoe, dress shoe,flip-flops, hosiery, sandal, shoe, spats, slipper, sock, or stocking.Further, a wearable device may be releasably or permanently attach to,or be part of, an accessory such as a bandana, belt, bow tie, coinpurse, cufflink, cummerbund, gaiters, glasses, gloves, headband,handbag, handkerchief, jewellery, muff, necktie, pocket protector,pocketwatch, sash, scarf, sunglasses, suspenders, umbrella, wallet, orwristwatch.

A wearable device may be releasably or permanently attach to, or be partof, an outwear such as an apron, blazer, British warm, cagoule, cape,chesterfield, coat, covert coat, cut-off, duffle coat, flight jacket,gilet, goggle jacket, guards coat, Harrington jacket, hoodie, jacket,leather jacket, mess jacket, opera coat, overcoat, parka, paletot, peacoat, poncho, raincoat, robe, safari jacket, shawl, shrug, ski suit,sleeved blanket, smoking jacket, sport coat, trench coat, ulster coat,waistcoat, or windbreaker. Similarly, a wearable device may bereleasably or permanently attach to, or be part of, a suit (or uniform)such as an academic dress, ball dress, black tie, boilersuit, cleanroomsuit, clerical clothing, court dress, gymslip, jumpsuit, kasaya, labcoat, military uniform, morning dress, onesie, pantsuit, red sea rig,romper suit, school uniform, scrubs, stroller, tuxedo, or white tie.Further, a wearable device may be releasably or permanently attach to,or be part of, a dress such as a ball gown, bouffant gown, coatdress,cocktail dress, debutante dress, formal wear, frock, evening gown, gown,house dress, jumper, little black dress, princess line, sheath dress,shirtdress, slip dress, strapless dress, sundress, wedding dress, orwrap dress. Furthermore, a wearable device may be releasably orpermanently attach to, or be part of, a skirt such as an A-line skirt,ballerina skirt, denim skirt, men's skirts, miniskirt, pencil skirt,prairie skirt, rah-rah skirt, sarong, Skort, tutu, or wrap. In oneexample, a wearable device may be releasably or permanently attach to,or be part of, a trousers (or shorts) such as bell-bottoms, bermudashorts, bondage pants, capri pants, cargo pants, chaps, cycling shorts,dress pants, high water pants, lowrise pants, Jeans, jodhpurs, leggings,overall, Palazzo pants, parachute pants, pedal pushers, phat pants,shorts, slim-fit pants, sweatpants, windpants, or yoga pants. In oneexample, a wearable device may be releasably or permanently attach to,or be part of, a top such as a blouse, crop top, dress shirt, guayabera,guernsey, halterneck, henley shirt, hoodie, jersey, polo shirt, shirt,sleeveless shirt, sweater, sweater vest, t-shirt, tube top, turtleneck,or twinset.

A wearable device may be structured, designed, or have a form factorthat is identical to, substantially similar to, or is at least in partsubstitute to, a fashion accessory. These accessories may be purelydecorative, or have a utility beyond aesthetics. Examples of theseaccessories include, but are not limited to, rings, bracelets,necklaces, watches, watch bands, purses, wallets, earrings, body rings,headbands, glasses, belts, ties, tie bars, tie tacks, wallets, shoes,pendants, charms and bobbles. For example, wearable devices may also beincorporated into pockets, steering wheels, keyboards, pens, and bicyclehandles.

In one example, the wearable device may be shaped as, or integratedwith, a device that includes an annular member defining an aperturetherethrough that is sized for receipt therein of a human body part. Thebody part may be part of a human hand such as upper arm, elbow, forearm,wrist (such as a wrist-band), or a finger (such as a ring).Alternatively or in addition, the body part may be part of a human heador neck, such as a forehead, ear, skull, or face. Alternatively or inaddition, the body part may be part of a human thorax or abdomen, suchas waist or hip. Alternatively or in addition, the body part may be partof a human leg or foot, such as thigh, calf, ankle, instep, knee, ortoe.

In one example, the wearable device may be shaped as, or integratedwith, a ring. The ring may comprise, consist essentially of or consistof a shank, which is the location that provides an opening for a finger,and a head, which comprises, consists essentially or consists ofornamental features of the ring and in some embodiments houses thesignaling assembly of the present device. The head may be of any shape,e.g., a regular sphere, truncated sphere, cube, rectangular prism,cylinder, triangular prism, cone, pyramid, barrel, truncated cone, domedcylinder, truncated cylinder, ellipsoid, regular polygon prism ortruncated three-dimensional polygon of e.g., 4-16 sides, such as atruncated pyramid (trapezoid), or combination thereof or it may be anirregular shape. Further, the head may comprise an upper face thatcontains and is configured to show one or more jewels and/or ornamentaldesigns.

A mobile communication device configured to be worn on an index fingerof a user's hand is described in U.S. Patent Application Publication No.2015/0373443 to Carroll entitled: “Finger-wearable mobile communicationdevice”, which is incorporated in its entirety for all purposes as iffully set forth herein. The device includes a case, a microphone, aswitch, and a power source. The microphone and the switch arestrategically located along a shape of the case so that as worn on theuser's index finger and when the switch is activated by the thumb of theuser's hand, the hand naturally cups about the microphone to form abarrier to ambient noise. Further, the microphone can readily be locatednear a corner of the user's mouth for optimal speech-receivingconditions and to provide more private audio input.

A user controls an external electronic device with a finger-ring-mountedtouchscreen is described in U.S. Patent Application Publication No.2015/0277559 to Vescovi et al. entitled: “Devices and Methods for a RingComputing Device”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The device includes a computerprocessor, wireless transceiver, and rechargeable power source; the ringis worn on a first finger receives an input from a second finger,selects one of a plurality of touch events associated with the input,and wirelessly transmits a command associated with the touch event tothe external electronic device.

A mobile communication device that comprises a fashion accessory and asignaling assembly is described in U.S. Patent Application PublicationNo. 2015/0349556 to Mercando et al. entitled: “Mobile CommunicationDevices”, which is incorporated in its entirety for all purposes as iffully set forth herein. The signaling assembly may be configured toprovide sensory stimuli such as a flashing LED light and a vibration.These stimuli may vary depending on the signal received from a remotecommunication device or from gestures made by a user or from informationstored in the mobile communication device.

A wearable fitness-monitoring device is described in U.S. Pat. No.8,948,832 to Hong et al. entitled: “Wearable Heart Rate Monitor”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The device including a motion sensor and a photoplethysmographic(PPG) sensor. The PPG sensor includes (i) a periodic light source, (ii)a photo detector, and (iii) circuitry determining a user's heart ratefrom an output of the photo detector. Some embodiments provide methodsfor operating a heart rate monitor of a wearable fitness-monitoringdevice to measure one or more characteristics of a heartbeat waveform.Some embodiments provide methods for operating the wearable fitnessmonitoring device in a low power state when the device determines thatthe device is not worn by a user. Some embodiments provide methods foroperating the wearable fitness-monitoring device in a normal power statewhen the device determines that the device is worn by a user.

A wearable device and method for processing mages to prolong batterylife are described in U.S. Pat. No. 8,957,988 to Wexler et al. entitled:“Apparatus for processing images to prolong battery life”, which isincorporated in its entirety for all purposes as if fully set forthherein. In one implementation, a wearable apparatus may include awearable image sensor configured to capture a plurality of images froman environment of a user. The wearable apparatus may also include atleast one processing device configured to, in a first processing-mode,process representations of the plurality of images to determine a valueof at least one capturing parameter for use in capturing at least onesubsequent image, and in a second processing-mode, process therepresentations of the plurality of images to extract information. Inaddition, the at least one processing device may operate in the firstprocessing-mode when the wearable apparatus is powered by a mobile powersource included in the wearable apparatus and may operate in the secondprocessing-mode when the wearable apparatus is powered by an externalpower source.

A wearable device may be used for notifying a person, such as by usingtactile, visual, or audible stimulus, as described for example in U.S.Patent Application No. 2015/0341901 to RYU et al. entitled: “Method andapparatus for providing notification”, which is incorporated in itsentirety for all purposes as if fully set forth herein, describing anelectronic device that includes: a transceiver configured to communicatewith at least one wearable device and receive, from the at least onewearable device, status information indicating whether the at least onewearable device is currently being worn; and a processor configured todetermine whether to send a notification request to the at least onewearable device based on the status information received by thetransceiver.

A communication device, system and method are described for example inU.S. Patent Application No. 2007/0052672 to Ritter et al. entitled:“Communication device, system and method”, which is incorporated in itsentirety for all purposes as if fully set forth herein. It is disclosescomprising a Virtual Retinal Display (VRD) in form of glasses (1), atleast one haptic sensor (12) mounted on the frame of said glasses orconnected by a short range communication interface (13) to said glasses(1), wherein it is possible to navigate by means of a cursor through animage displayed by the Virtual Retinal Display (VRD) with the at leastone haptic sensor (12). A central control unit controls (11) the VirtualRetinal Display (VRD) and the at least one haptic sensor (12). When theVirtual Retinal Display (VRD) is connected to an external device (2, 9)by a short range communication interface (13), the user can navigatethrough the content of the external device (2, 9) by easy use of thehaptic sensor (12).

Wearable communication devices, e.g. implemented in a watch, using shortrange communication to a cell phone, and facilitating natural andintuitive user interface with low-power implementation are described forexample in U.S. Patent Application No. 2014/0045547 to Singamsetty etal. entitled: “Wearable Communication Device and User Interface”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The devices allow a user to easily access all features of thephone, all while a phone is nearby but not visible. Notification isperformed with vibration, an LED light and OLED text display of incomingcalls, texts, and calendar events. It allows communicating hands-free.This allows using the communication device as “remote control” for homedevices, etc. via voice and buttons. The device comprises interfacesmotion sensors such as accelerometers, magnetometer and gyroscope,infrared proximity sensors, vibrator motor, and/or voice recognition.Low power consumption is achieved by dynamical configuration of sensorparameters to support only the necessary sensor functions at any givenstate of the device.

A wearable electronic device that is configured to control and command avariety of wireless devices within its proximity is described in U.S.Pat. No. 7,605,714 to Thompson et al. entitled: “System and method forcommand and control of wireless devices using a wearable device”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The wearable device dynamically generates a user interfacecorresponding to the services of a particular wireless device. Throughthe user interface, the wireless device surface content to a user andallows a user select interactions with the wireless devices using thewearable device.

An apparatus and method for the remote control and/or interaction-withelectronic-devices such as computers; home-entertainment-systems;media-centers; televisions; DVD-players; VCR-players; music systems;appliances; security systems; toys/games; and/or displays are describedin U.S. Pat. No. 8,508,472 to Wieder entitled: “Wearable remote controlwith a single control button”, which is incorporated in its entirety forall purposes as if fully set forth herein. A user may orient a pointer(e.g., laser pointer) to place a pointer-spot on/near object(s) on anactive-display(s); and/or a fixed-display(s); and/or on real-worldobject(s) within a display region or pointer-spot detection-region.Detectors, imager(s) and/or camera(s) may be connected/attached to thedisplay region and/or a structure that is connected/attached to displayregion. When the user initiates a “select”, the detectors/cameras maydetect the location of the pointer-spot within the display region.Corresponding to the user's selection(s); control action(s) may beperformed on the device(s) being controlled/interacted-with andadditional selection-menus may be optionally presented on anactive-display.

A hand-worn controller consisting of a housing having a central openingsized to permit the controller to be worn as ring on the index finger ofa human hand is described in U.S. Patent Application Publication No.2006/0164383 to Machin et al. entitled: “Remote controller ring for userinteraction”, which is incorporated in its entirety for all purposes asif fully set forth herein. A joystick lever projects outwardly from saidhousing and is positioned to be manipulated by the user's thumb. Thejoystick operates on or more control devices, such as switches orpotentiometers, that produce control signals. A wireless communicationsdevice, such as a Bluetooth module, mounted in said housing transmitscommand signals to a remote utilization device, which are indicative ofthe motion or position of said joystick lever.

A wearable augmented reality computing apparatus with a display screen,a reflective device, a computing device and a head mounted harness tocontain these components is described in U.S. Patent ApplicationPublication No. 2012/0050144 to Morlock entitled: “Wearable augmentedreality computing apparatus”, which is incorporated in its entirety forall purposes as if fully set forth herein. The display device andreflective device are configured such that a user can see the reflectionfrom the display device superimposed on the view of reality. Anembodiment uses a switchable mirror as the reflective device. One usageof the apparatus is for vehicle or pedestrian navigation. The portabledisplay and general purpose computing device can be combined in a devicesuch as a smartphone. Additional components consist of orientationsensors and non-handheld input devices.

In one example, a wearable device may use, or may be based on, aprocessor or a microcontroller that is designed for wearableapplications, such as the CC2650 SimpleLink™ Multistandard Wireless MCUavailable from Texas Instruments Incorporated (headquartered in Dallas,Tex., U.S.A.) and described in a Texas Instrument 2015 publication#SWRT022 entitled: “SimpleLink™ Ultra-Low Power—Wireless MicrocontrollerPlatform”, and in a Texas Instrument 2015 datasheet #SWRS158A (publishedFebruary 2015, Revised October 2015) entitled: “CC2650 SimpleLink™Multistandard Wireless MCU”, which are both incorporated in theirentirety for all purposes as if fully set forth herein.

An example of a personal multimedia electronic device, and moreparticularly to a head-worn device such as an eyeglass frame, isdescribed in U.S. Patent Application No. 2010/0110368 to Chaum entitled:“System and apparatus for eyeglass appliance platform”, which isincorporated in its entirety for all purposes as if fully set forthherein. The device is having a plurality of interactiveelectrical/optical components. In one embodiment, a personal multimediaelectronic device includes an eyeglass frame having a side arm and anoptic frame; an output device for delivering an output to the wearer; aninput device for obtaining an input; and a processor comprising a set ofprogramming instructions for controlling the input device and the outputdevice. The output device is supported by the eyeglass frame and isselected from the group consisting of a speaker, a bone conductiontransmitter, an image projector, and a tactile actuator. The inputdevice is supported by the eyeglass frame and is selected from the groupconsisting of an audio sensor, a tactile sensor, a bone conductionsensor, an image sensor, a body sensor, an environmental sensor, aglobal positioning system receiver, and an eye tracker. In oneembodiment, the processor applies a user interface logic that determinesa state of the eyeglass device and determines the output in response tothe input and the state.

An example of an eyewear for a user is described in U.S. PatentApplication No. 2012/0050668 Howell et al. entitled: “Eyewear withtouch-sensitive input surface”, which is incorporated in its entiretyfor all purposes as if fully set forth herein. The eyewear includes aneyewear frame, electrical circuitry at least partially in the eyewearframe, and a touch sensitive input surface on the eyewear frameconfigured to provide an input to the electrical circuitry to perform afunction via touching the touch sensitive input surface. In anotherembodiment, the eyewear includes a switch with at least two operationalstates. The operational states of the switch can be configured to bechanged by sliding a finger across the touch sensitive input surface ofthe frame.

An example of a wearable computing device is described in U.S. PatentApplication No. 2013/0169513 to Heinrich et al. entitled: “Wearablecomputing device”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The device includes a boneconduction transducer, an extension arm, a light pass hole, and aflexible touch pad input circuit. When a user wears the device, thetransducer contacts the user's head. A display is attached to a free endof an extension arm. The extension arm is pivotable such that a distancebetween the display and the user's eye is adjustable to provide thedisplay at an optimum position. The light pass hole may include a lightemitting diode and a flash. The touch pad input circuit may be adheredto at least one side arm such that parting lines are not providedbetween edges of the circuit and the side arm.

Speech synthesis. A speech synthesizer is used to produce natural andintelligible artificial human speech may be implemented in hardware, insoftware, or combination thereof. A speech synthesizer may beText-To-Speech (TTS) based, that converts normal language text tospeech, or alternatively (or in addition) may be based on renderingsymbolic linguistic representation like phonetic transcription. A TTStypically involves two steps, the front-end where the raw input text ispre-processed to fully write-out words replacing numbers andabbreviations, followed by assigning phonetic transcriptions to eachword (text-to-phoneme), and the back-end (or synthesizer) where thesymbolic linguistic representation is converted to output sound.

The generating of synthetic speech waveform typically uses aconcatenative or formant synthesis. The concatenative synthesis commonlyproduces the most natural-sounding synthesized speech, and is based onthe concatenation (or stringing together) of segments of recordedspeech. There are three main types of concatenative synthesis: Unitselection, diphone synthesis, and domain-specific synthesis. Unitselection synthesis is based on large databases of recorded speechincluding individual phones, diphones, half-phones, syllables,morphemes, words, phrases, and sentences, indexed based on thesegmentation and acoustic parameters like the fundamental frequency(pitch), duration, position in the syllable, and neighboring phones. Atrun time, the desired target utterance is created by determining(typically using a specially weighted decision tree) the best chain ofcandidate units from the database (unit selection). Diphone synthesisuses a minimal speech database containing all the diphones(sound-to-sound transitions) occurring in a language, and at runtime,the target prosody of a sentence is superimposed on these minimal unitsby means of digital signal processing techniques such as linearpredictive coding. Domain-specific synthesis is used where the output islimited to a particular domain, using concatenated prerecorded words andphrases to create complete utterances. In formant synthesis thesynthesized speech output is created using additive synthesis and anacoustic model (physical modeling synthesis), rather than on using humanspeech samples. Parameters such as fundamental frequency, voicing, andnoise levels are varied over time to create a waveform of artificialspeech. The synthesis may further be based on articulatory synthesiswhere computational techniques for synthesizing speech are based onmodels of the human vocal tract and the articulation processes occurringthere, or may be HMM-based synthesis which is based on hidden Markovmodels, where the frequency spectrum (vocal tract), fundamentalfrequency (vocal source), and duration (prosody) of speech are modeledsimultaneously by HMMs and generated based on the maximum likelihoodcriterion. The speech synthesizer may further be based on the bookentitled: “Development in Speech Synthesis”, by Mark Tatham andKatherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN:0-470-85538-X, on the book entitled: “Speech Synthesis and Recognition”by John Holmes and Wendy Holmes, 2^(nd) Edition, published 2001 ISBN:0-7484-0856-8, on the book entitled: “Techniques and Challenges inSpeech Synthesis—Final Report” by David Ferris [ELEC4840B] publishedApr. 11, 2016, and on the book entitled: “Text-to-Speech Synthesis” byPaul Taylor [ISBN 978-0-521-89927-7] published 2009 by CambridgeUniversity Press, which are all incorporated in their entirety for allpurposes as if fully set forth herein.

A speech synthesizer may be software-based such as Apple VoiceOverutility which uses speech synthesis for accessibility, and is part ofthe Apple iOS operating system used on the iPhone, iPad and iPod Touch.Similarly, Microsoft uses SAPI 4.0 and SAPI 5.0 as part of Windowsoperating system. A speech synthesizer may be hardware based, such asbased on Sensory Inc. NLP-5x described in the Data sheet “NaturalLanguage Processor with Motor, Sensor and Display Control”, P/N80-0317-K, published 2010 by Sensory, Inc. of Santa-Clara, Calif.,U.S.A., which is incorporated herein in its entirety for all purposes asif fully set forth herein.

In one example, the sounder may be used to play a stored digital audio.The digital audio content can be stored in the sounder. Further, fewfiles may be stored (e.g., representing different announcements orsongs), selected by the control logic. Alternatively or in addition, thedigital audio data may be received by the sounder from external sourcesvia any of the above networks. Furthermore, the source of the digitalaudio may be a microphone serving as a sensor, either after processing,storing, delaying, or any other manipulation, or as originally receivedresulting ‘doorphone’ or ‘intercom’ functionality between a microphoneand a sounder in the building.

In another example, the sounder simulates the voice of a human being orgenerates music, typically by using an electronic circuit having amemory for storing the sounds (e.g., music, song, voice message, etc.),a digital to analog converter 62 to reconstruct the electricalrepresentation of the sound, and a driver for driving a loudspeaker,which is an electro-acoustic transducer that converts an electricalsignal to sound. An example of a greeting card providing music andmechanical movement is disclosed in U.S. Patent Application No.2007/0256337 to Segan entitled: “User Interactive Greeting Card”, whichis incorporated in its entirety for all purposes as if fully set forthherein.

In one example, the system is used for sound or music generation. Forexample, the sound produced can emulate the sounds of a conventionalacoustical music instrument, such as a piano, tuba, harp, violin, flute,guitar and so forth. In one example, the sounder is an audible signalingdevice, emitting audible sounds that can be heard (having frequencycomponents in the 20-20,000 Hz band). In one example the sound generatedis music or song. The elements of the music such as pitch (which governsmelody and harmony), rhythm (and its associated concepts tempo, meter,and articulation), dynamics, and the sonic qualities of timbre andtexture, may be associated with the shape theme. For example, if amusical instrument shown in the picture, the music generated by thatinstrument will be played, e.g., drumming sound of drums and playing ofa flute or guitar. In one example, a talking human voice is played bythe sounder. The sound may be a syllable, a word, a phrase, a sentence,a short story or a long story, and can be based on speech synthesis orprerecorded. Male or female voice can be used, further being young orold.

Some examples of toys that include generation of an audio signal such asmusic are disclosed in U.S. Pat. No. 4,496,149 to Schwartzberg entitled:“Game Apparatus Utilizing Controllable Audio Signals”, in U.S. Pat. No.4,516,260 to Breedlove et al. entitled: “Electronic Learning Aid or Gamehaving Synthesized Speech”, in U.S. Pat. No. 7,414,186 to Scarpa et al.entitled: “System and Method for Teaching Musical Notes”, in U.S. Pat.No. 4,968,255 to Lee et al., entitled: “Electronic InstructionalApparatus”, in U.S. Pat. No. 4,248,123 to Bunger et al., entitled:“Electronic Piano” and in U.S. Pat. No. 4,796,891 to Milner entitled:“Musical Puzzle Using Sliding Tiles”, and toys with means forsynthesizing human voice are disclosed in U.S. Pat. No. 6,527,611 toCummings entitled: “Place and Find Toy”, and in U.S. Pat. No. 4,840,602to Rose entitled: “Talking Doll Responsive to External Signal”, whichare all incorporated in their entirety for all purposes as if fully setforth herein. A music toy kit combining music toy instrument with a setof construction toy blocks is disclosed in U.S. Pat. No. 6,132,281 toKlitsner et al. entitled: “Music Toy Kit” and in U.S. Pat. No. 5,349,129to Wisniewski et al. entitled: “Electronic Sound Generating Toy”, whichare incorporated in their entirety for all purposes as if fully setforth herein.

Database. A database is an organized collection of data, typicallymanaged by a DataBase Management System (DBMS) that organizes thestorage of data and performs other functions such as the creation,maintenance, and usage of the database storage structures. The data istypically organized to model aspects of reality in a way that supportsprocesses requiring information. Databases commonly also provide userswith a user interface and front-end that enables the users to query thedatabase, often in complex manners that require processing andorganization of the data. The term “database” is used herein to refer toa database, or to both a database and the DBMS used to manipulate it.Database Management Systems (DBMS) are typically computer softwareapplications that interact with the user, other applications, and thedatabase itself to capture and analyze data, typically providing variousfunctions that allow entry, storage and retrieval of large quantities ofinformation, as well as providing ways to manage how that information isorganized. A general-purpose DBMS is designed to allow the definition,creation, querying, update, and administration of databases. Examples ofDBMSs include MySQL, PostgreSQL, Microsoft SQL Server, Oracle, Sybaseand IBM DB2. Database technology and application is described in adocument published by Telemark University College entitled “Introductionto Database Systems”, authored by Hans-Petter Halvorsen (dated 2014 Mar.3), which is incorporated in its entirety for all purposes as if fullyset forth herein.

SQL. Structured Query Language (SQL) is a widely-used programminglanguage for working with relational databases, designed for managingdata held in a relational database management system (RDBMS), or forstream processing in a relational data stream management system(RDSMS).SQL consists of a data definition language and a datamanipulation language. The scope of SQL includes data insert, query,update and delete, schema creation and modification, and data accesscontrol. Although SQL is often described as, and largely is, adeclarative language (4GL), it also includes procedural elements. SQL isdesigned for querying data contained in a relational database, and is aset-based, declarative query language. The SQL is standardized asISO/IEC 9075:2011 standard: “Information technology—Databaselanguages—SQL”. The ISO/IEC 9075 standard is complemented by ISO/IEC13249 standard: “SQL Multimedia and Application Packages” that definesinterfaces and packages based on SQL. The aim is a unified access totypical database applications like text, pictures, data mining orspatial data. SQL is described in the tutorial entitled: “Oracle/SQLTutorial” by Michael Gertz of University of California, which isincorporated in its entirety for all purposes as if fully set forthherein.

DSP. A Digital Signal Processor (DSP) is a specialized microprocessor(or a SIP block), with its architecture optimized for the operationalneeds of digital signal processing, serving the goal of DSPs is usuallyto measure, filter and/or compress continuous real-world analog signals.Most general-purpose microprocessors can also execute digital signalprocessing algorithms successfully, but dedicated DSPs usually havebetter power efficiency thus they are more suitable in portable devicessuch as mobile phones because of power consumption constraints. DSPsoften use special memory architectures that are able to fetch multipledata and/or instructions at the same time. Digital signal processingalgorithms typically require a large number of mathematical operationsto be performed quickly and repeatedly on a series of data samples.Signals (perhaps from audio or video sensors) are constantly convertedfrom analog to digital, manipulated digitally, and then converted backto analog form. Many DSP applications have constraints on latency; thatis, for the system to work, the DSP operation must be completed withinsome fixed time, and deferred (or batch) processing is not viable. Aspecialized digital signal processor, however, will tend to provide alower-cost solution, with better performance, lower latency, and norequirements for specialized cooling or large batteries. Thearchitecture of a digital signal processor is optimized specifically fordigital signal processing. Most also support some of the features as anapplications processor or microcontroller, since signal processing israrely the only task of a system. Some useful features for optimizingDSP algorithms are outlined below.

Hardware features visible through DSP instruction sets commonly includehardware modulo addressing, allowing circular buffers to be implementedwithout having to constantly test for wrapping; a memory architecturedesigned for streaming data, using DMA extensively and expecting code tobe written to know about cache hierarchies and the associated delays;driving multiple arithmetic units may require memory architectures tosupport several accesses per instruction cycle; separate program anddata memories (Harvard architecture), and sometimes concurrent access onmultiple data buses; and specianalyzeal SIMD (single instruction,multiple data) operations. Digital signal processing is furtherdescribed in a book by John G. Proakis and Dimitris G. Manolakis,published 1996 by Prentice-Hall Inc. [ISBN 0-13-394338-9] entitled:“Third Edition—DIGITAL SIGNAL PROCESSING—Principles, Algorithms, andApplication”, and in a book by Steven W. Smith entitled: “The Scientistand Engineer's Guide to—Digital Signal Processing—Second Edition”,published by California Technical Publishing [ISBN 0-9960176-7-6], whichare both incorporated in their entirety for all purposes as if fully setforth herein.

ANN. Neural networks (or Artificial Neural Networks (ANNs)) are a familyof statistical learning models inspired by biological neural networks(the central nervous systems of animals, in particular the brain) andare used to estimate or approximate functions that may depend on a largenumber of inputs and are generally unknown. Artificial neural networksare generally presented as systems of interconnected “neurons” whichsend messages to each other. The connections have numeric weights thatcan be tuned based on experience, making neural nets adaptive to inputsand capable of learning. For example, a neural network for handwritingrecognition is defined by a set of input neurons that may be activatedby the pixels of an input image. After being weighted and transformed bya function (determined by the network designer), the activations ofthese neurons are then passed on to other neurons, and this process isrepeated until finally, an output neuron is activated, and determineswhich character was read. Like other machine learning methods—systemsthat learn from data—neural networks have been used to solve a widevariety of tasks that are hard to solve using ordinary rule-basedprogramming, including computer vision and speech recognition. A classof statistical models is typically referred to as “Neural” if itcontains sets of adaptive weights, i.e. numerical parameters that aretuned by a learning algorithm, and capability of approximatingnon-linear functions from their inputs. The adaptive weights can bethought of as connection strengths between neurons, which are activatedduring training and prediction. Neural Networks are described in a bookby David Kriesel entitled: “A Brief Introduction to Neural Networks”(ZETA2-EN) [downloaded May 2015 from www.dkriesel.com], which isincorporated in its entirety for all purposes as if fully set forthherein. Neural Networks are further described in a book by Simon Haykinpublished 2009 by Pearson Education, Inc. [ISBN—978-0-13-147139-9]entitled: “Neural Networks and Learning Machines—Third Edition”, whichis incorporated in its entirety for all purposes as if fully set forthherein.

Neural networks based techniques may be used for image processing, asdescribed in an article in Engineering Letters, 20:1, EL_20_1_09(Advance online publication: 27 Feb. 2012) by Juan A. Ramirez-Quintana,Mario I. Cacon-Murguia, and F. Chacon-Hinojos entitled: “ArtificialNeural Image Processing Applications: A Survey”, in an article published2002 by Pattern Recognition Society in Pattern Recognition 35 (2002)2279-2301 [PII: S0031-3203(01)00178-9] authored by M. Egmont-Petersen,D. de Ridder, and H. Handels entitled: “Image processing with neuralnetworks—a review”, and in an article by Dick de Ridder et al. (of theUtrecht University, Utrecht, The Netherlands) entitled: “Nonlinear imageprocessing using artificial neural networks”, which are all incorporatedin their entirety for all purposes as if fully set forth herein.

Neural networks may be used for object detection as described in anarticle by Christian Szegedy, Alexander Toshev, and Dumitru Erhan (ofGoogle, Inc.) (downloaded July 2015) entitled: “Deep Neural Networks forObject Detection”, in a CVPR2014 paper provided by the Computer VisionFoundation by Dumitru Erhan, Christian Szegedy, Alexander Toshev, andDragomir Anguelov (of Google, Inc., Mountain-View, Calif., U.S.A.)(downloaded July 2015) entitled: “Scalable Object Detection using DeepNeural Networks”, and in an article by Shawn McCann and Jim Reesman(both of Stanford University) (downloaded July 2015) entitled: “ObjectDetection using Convolutional Neural Networks”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

Using neural networks for object recognition or classification isdescribed in an article (downloaded July 2015) by Mehdi Ebady Manaa,Nawfal Turki Obies, and Dr. Tawfiq A. Al-Assadi (of Department ofComputer Science, Babylon University), entitled: “Object Classificationusing neural networks with Gray-level Co-occurrence Matrices (GLCM)”, ina technical report No. IDSIA-01-11 Jan. 2001 published byIDSIA/USI-SUPSI and authored by Dan C. Ciresan et al. entitled:“High-Performance Neural Networks for Visual Object Classification”, inan article by Yuhua Zheng et al. (downloaded July 2015) entitled:“Object Recognition using Neural Networks with Bottom-Up and top-DownPathways”, and in an article (downloaded July 2015) by Karen Simonyan,Andrea Vedaldi, and Andrew Zisserman (all of Visual Geometry Group,University of Oxford), entitled: “Deep Inside Convolutional Networks:Visualising Image Classification Models and Saliency Maps”, which areall incorporated in their entirety for all purposes as if fully setforth herein.

Using neural networks for object recognition or classification isfurther described in U.S. Pat. No. 6,018,728 to Spence et al. entitled:“Method and Apparatus for Training a Neural Network to LearnHierarchical Representations of Objects and to Detect and ClassifyObjects with Uncertain Training Data”, in U.S. Pat. No. 6,038,337 toLawrence et al. entitled: “Method and Apparatus for Object Recognition”,in U.S. Pat. No. 8,345,984 to Ji et al. entitled: “3D ConvolutionalNeural Networks for Automatic Human Action Recognition”, and in U.S.Pat. No. 8,705,849 to Prokhorov entitled: “Method and System for ObjectRecognition Based on a Trainable Dynamic System”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

Actual ANN implementation may be based on, or may use, the MATLAB® ANNdescribed in the User's Guide Version 4 published July 2002 by TheMathWorks, Inc. (Headquartered in Natick, Mass., U.S.A.) entitled:“Neural Network ToolBox—For Use with MATLAB®” by Howard Demuth and MarkBeale, which is incorporated in its entirety for all purposes as iffully set forth herein. A VHDL IP core that is a configurablefeedforward Artificial Neural Network (ANN) for implementation in FPGAsis available (under the Name: artificial_neural_network, created Jun. 2,2016 and updated Oct. 11, 2016) from OpenCores organization,downloadable from http://opencores.org/. This IP performs fullfeedforward connections between consecutive layers. All neurons' outputsof a layer become the inputs for the next layer. This ANN architectureis also known as Multi-Layer Perceptron (MLP) when is trained with asupervised learning algorithm. Different kinds of activation functionscan be added easily coding them in the provided VHDL template. This IPcore is provided in two parts: kernel plus wrapper. The kernel is theoptimized ANN with basic logic interfaces. The kernel should beinstantiated inside a wrapper to connect it with the user's systembuses. Currently, an example wrapper is provided for instantiate it onXilinx Vivado, which uses AXI4 interfaces for AM BA buses.

Dynamic neural networks are the most advanced in that they dynamicallycan, based on rules, form new connections and even new neural unitswhile disabling others. In a feedforward neural network (FNN), theinformation moves in only one direction—forward: From the input nodesdata goes through the hidden nodes (if any) and to the output nodes.There are no cycles or loops in the network. Feedforward networks can beconstructed from different types of units, e.g. binary McCulloch-Pittsneurons, the simplest example being the perceptron. Contrary tofeedforward networks, Recurrent Neural Networks (RNNs) are models withbi-directional data flow. While a feedforward network propagates datalinearly from input to output, RNNs also propagate data from laterprocessing stages to earlier stages. RNNs can be used as generalsequence processors.

Any ANN herein may be based on, may use, or may be trained or used,using the schemes, arrangements, or techniques described in the book byDavid Kriesel entitled: “A Brief Introduction to Neural Networks”(ZETA2-EN) [downloaded May 2015 from www.dkriesel.com], in the book bySimon Haykin published 2009 by Pearson Education, Inc.[ISBN—978-0-13-147139-9] entitled: “Neural Networks and LearningMachines—Third Edition”, in the article in Engineering Letters, 20:1,EL_20_1_09 (Advance online publication: 27 Feb. 2012) by Juan A.Ramirez-Quintana, Mario I. Cacon-Murguia, and F. Chacon-Hinojosentitled: “Artificial Neural Image Processing Applications: A Survey”,or in the article entitled: “Image processing with neural networks—areview”, and in the article by Dick de Ridder et al. (of the UtrechtUniversity, Utrecht, The Netherlands) entitled: “Nonlinear imageprocessing using artificial neural networks”.

Any object detection herein using ANN may be based on, may use, or maybe trained or used, using the schemes, arrangements, or techniquesdescribed in the article by Christian Szegedy, Alexander Toshev, andDumitru Erhan (of Google, Inc.) entitled: “Deep Neural Networks forObject Detection”, in the CVPR2014 paper provided by the Computer VisionFoundation entitled: “Scalable Object Detection using Deep NeuralNetworks”, in the article by Shawn McCann and Jim Reesman entitled:“Object Detection using Convolutional Neural Networks”, or in any otherdocument mentioned herein.

Any object recognition or classification herein using ANN may be basedon, may use, or may be trained or used, using the schemes, arrangements,or techniques described in the article by Mehdi Ebady Manaa, NawfalTurki Obies, and Dr. Tawfiq A. Al-Assadi entitled: “ObjectClassification using neural networks with Gray-level Co-occurrenceMatrices (GLCM)”, in the technical report No. IDSIA-01-11 entitled:“High-Performance Neural Networks for Visual Object Classification”, inthe article by Yuhua Zheng et al. entitled: “Object Recognition usingNeural Networks with Bottom-Up and top-Down Pathways”, in the article byKaren Simonyan, Andrea Vedaldi, and Andrew Zisserman, entitled: “DeepInside Convolutional Networks: Visualising Image Classification Modelsand Saliency Maps”, or in any other document mentioned herein.

A logical representation example of a simple feed-forward ArtificialNeural Network (ANN) is shown in FIG. 4 . The ANN 40 provides threeinputs designated as IN #1 42 a, IN #2 42 b, and IN #3 42 c, whichconnects to three respective neuron units forming an input layer 41 a.Each neural unit is linked some of, or to all of, a next layer 41 b,with links that may be enforced or inhibit by associating weights aspart of the training process. An output layer 41 d consists of twoneuron units that feeds two outputs OUT #1 43 a and OUT #2 43 b. Anotherlayer 41 c is coupled between the layer 41 b and the output layer 41 d.The intervening layers 41 b and 41 c are referred to as hidden layers.While three inputs are exampled in the ANN 40, any number of inputs maybe equally used, and while two output are exampled in the ANN 40, anynumber of outputs may equally be used. Further, the ANN 40 uses fourlayers, consisting of an input layer, an output layer, and two hiddenlayers. However, any number of layers may be used. For example, thenumber of layers may be equal to, or above than, 3, 4, 5, 7, 10, 15, 20,25, 30, 35, 40, 45, or 50 layers. Similarly, an ANN may have any numberbelow 4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers.

DNN. A Deep Neural Network (DNN) is an artificial neural network (ANN)with multiple layers between the input and output layers. For example, aDNN that is trained to recognize dog breeds will go over the given imageand calculate the probability that the dog in the image is a certainbreed. The user can review the results and select which probabilitiesthe network should display (above a certain threshold, etc.) and returnthe proposed label. Each mathematical manipulation as such is considereda layer, and complex DNN have many layers, hence the name “deep”networks. DNNs can model complex non-linear relationships. DNNarchitectures generate compositional models where the object isexpressed as a layered composition of primitives. The extra layersenable composition of features from lower layers, potentially modelingcomplex data with fewer units than a similarly performing shallownetwork. Deep architectures include many variants of a few basicapproaches. Each architecture has found success in specific domains. Itis not always possible to compare the performance of multiplearchitectures, unless they have been evaluated on the same data sets.DNN is described in a book entitled: “Introduction to Deep Learning FromLogical Calculus to Artificial Intelligence” by Sandro Skansi [ISSN1863-7310 ISSN 2197-1781, ISBN 978-3-319-73003-5], published 2018 bySpringer International Publishing AG, which is incorporated in itsentirety for all purposes as if fully set forth herein.

Deep Neural Networks (DNNs), which employ deep architectures canrepresent functions with higher complexity if the numbers of layers andunits in a single layer are increased. Given enough labeled trainingdatasets and suitable models, deep learning approaches can help humansestablish mapping functions for operation convenience. In this paper,four main deep architectures are recalled and other methods (e.g. sparsecoding) are also briefly discussed. Additionally, some recent advancesin the field of deep learning are described. The purpose of this articleis to provide a timely review and introduction on the deep learningtechnologies and their applications. It is aimed to provide the readerswith a background on different deep learning architectures and also thelatest development as well as achievements in this area. The rest of thepaper is organized as follows. In Sections II-V, four main deep learningarchitectures, which are Restricted Boltzmann Machines (RBMs), DeepBelief Networks (DBNs), AutoEncoder (AE), and Convolutional NeuralNetworks (CNNs), are reviewed, respectively. Comparisons are made amongthese deep architectures and recent developments on these algorithms arediscussed. A schematic diagram 40 a of an RBM, a schematic diagram 40 bof a DBN, and a schematic structure 40 c of a CNN are shown in FIG. 4 a.

DNNs are typically feedforward networks in which data flows from theinput layer to the output layer without looping back. At first, the DNNcreates a map of virtual neurons and assigns random numerical values, or“weights”, to connections between them. The weights and inputs aremultiplied and return an output between 0 and 1. If the network did notaccurately recognize a particular pattern, an algorithm would adjust theweights. That way the algorithm can make certain parameters moreinfluential, until it determines the correct mathematical manipulationto fully process the data. Recurrent neural networks (RNNs), in whichdata can flow in any direction, are used for applications such aslanguage modeling. Long short-term memory is particularly effective forthis use. Convolutional deep neural networks (CNNs) are used in computervision. CNNs also have been applied to acoustic modeling for AutomaticSpeech Recognition (ASR).

Since the proposal of a fast learning algorithm for deep belief networksin 2006, the deep learning techniques have drawn ever-increasingresearch interests because of their inherent capability of overcomingthe drawback of traditional algorithms dependent on hand-designedfeatures. Deep learning approaches have also been found to be suitablefor big data analysis with successful applications to computer vision,pattern recognition, speech recognition, natural language processing,and recommendation systems.

Widely-used deep learning architectures and their practical applicationsare discussed in a paper entitled: “A Survey of Deep Neural NetworkArchitectures and Their Applications” by Weibo Liva, Zidong Wanga,Xiaohui Liva, Nianyin Zengb, Yurong Liuc, and Fuad E. Alsaadid,published December 2016 [DOI: 10.1016/j.neucom.2016.12.038] inNeurocomputing 234, which is incorporated in its entirety for allpurposes as if fully set forth herein. An up-to-date overview isprovided on four deep learning architectures, namely, autoencoder,convolutional neural network, deep belief network, and restrictedBoltzmann machine. Different types of deep neural networks are surveyedand recent progresses are summarized. Applications of deep learningtechniques on some selected areas (speech recognition, patternrecognition and computer vision) are highlighted. A list of futureresearch topics are finally given with clear justifications.

RBM. Restricted Boltzmann machine (RBM) is a generative stochasticartificial neural network that can learn a probability distribution overits set of inputs. As their name implies, RBMs are a variant ofBoltzmann machines, with the restriction that their neurons must form abipartite graph: a pair of nodes from each of the two groups of units(commonly referred to as the “visible” and “hidden” units respectively)may have a symmetric connection between them; and there are noconnections between nodes within a group. By contrast, “unrestricted”Boltzmann machines may have connections between hidden units. Thisrestriction allows for more efficient training algorithms than areavailable for the general class of Boltzmann machines, in particular thegradient-based contrastive divergence algorithm. Restricted Boltzmannmachines can also be used in deep learning networks. In particular, deepbelief networks can be formed by “stacking” RBMs and optionallyfine-tuning the resulting deep network with gradient descent andbackpropagation

DBN. A Deep Belief Network (DBN) is a generative graphical model, oralternatively a class of deep neural network, composed of multiplelayers of latent variables (“hidden units”), with connections betweenthe layers but not between units within each layer. When trained on aset of examples without supervision, a DBN can learn toprobabilistically reconstruct its inputs. The layers then act as featuredetectors. After this learning step, a DBN can be further trained withsupervision to perform classification. DBNs can be viewed as acomposition of simple, unsupervised networks such as restrictedBoltzmann machines (RBMs) or autoencoders, where each sub-network'shidden layer serves as the visible layer for the next. An RBM is anundirected, generative energy-based model with a “visible” input layerand a hidden layer and connections between but not within layers. Thiscomposition leads to a fast, layer-by-layer unsupervised trainingprocedure, where contrastive divergence is applied to each sub-networkin turn, starting from the “lowest” pair of layers (the lowest visiblelayer is a training set).

Dynamic neural networks are the most advanced in that they dynamicallycan, based on rules, form new connections and even new neural unitswhile disabling others. In a Feedforward Neural Network (FNN), theinformation moves in only one direction—forward: From the input nodesdata goes through the hidden nodes (if any) and to the output nodes.There are no cycles or loops in the network. Feedforward networks can beconstructed from different types of units, e.g. binary McCulloch-Pittsneurons, the simplest example being the perceptron. Contrary tofeedforward networks, Recurrent Neural Networks (RNNs) are models withbi-directional data flow. While a feedforward network propagates datalinearly from input to output, RNNs also propagate data from laterprocessing stages to earlier stages. RNNs can be used as generalsequence processors.

A waveform analysis assembly (10) that includes a sensor (12) fordetecting physiological electrical and mechanical signals produced bythe body is disclosed in U.S. Pat. No. 5,092,343 to Spitzer et al.entitled: “Waveform analysis apparatus and method using neural networktechniques”, which is incorporated in its entirety for all purposes asif fully set forth herein. An extraction neural network (22, 22′) willlearn a repetitive waveform of the electrical signal, store the waveformin memory (18), extract the waveform from the electrical signal, storethe location times of occurrences of the waveform, and subtract thewaveform from the electrical signal. Each significantly differentwaveform in the electrical signal is learned and extracted. A single ormultilayer layer neural network (22, 22′) accomplishes the learning andextraction with either multiple passes over the electrical signal oraccomplishes the learning and extraction of all waveforms in a singlepass over the electrical signal. A reducer (20) receives the storedwaveforms and times and reduces them into features characterizing thewaveforms. A classifier neural network (36) analyzes the features byclassifying them through non-linear mapping techniques within thenetwork representing diseased states and produces results of diseasedstates based on learned features of the normal and patient groups.

A real-time waveform analysis system that utilizes neural networks toperform various stages of the analysis is disclosed in U.S. Pat. No.5,751,911 to Goldman entitled: “Real-time waveform analysis usingartificial neural networks”, which is incorporated in its entirety forall purposes as if fully set forth herein. The signal containing thewaveform is first stored in a buffer and the buffer contents transmittedto a first and second neural network, which have been previously trainedto recognize the start point and the end point of the waveformrespectively. A third neural network receives the signal occurringbetween the start and end points and classifies that waveform ascomprising either an incomplete waveform, a normal waveform or one of avariety of predetermined characteristic classifications. Ambiguities inthe output of the third neural network are arbitrated by a fourth neuralnetwork, which may be given additional information, which serves toresolve these ambiguities. In accordance with the preferred embodiment,the present invention is applied to a system analyzing respiratorywaveforms of a patient undergoing anesthesia and the classifications ofthe waveform correspond to normal or various categories of abnormalfeatures functioning in the respiratory signal. The system performs theanalysis rapidly enough to be used in real-time systems and can beoperated with relatively low-cost hardware and with minimal softwaredevelopment required.

A method for analyzing data is disclosed in U.S. Pat. No. 8,898,093 toHelmsen entitled: “Systems and methods for analyzing data using deepbelief networks (DBN) and identifying a pattern in a graph”, which isincorporated in its entirety for all purposes as if fully set forthherein. The method includes generating, using a processing device, agraph from raw data, the graph including a plurality of nodes and edges,deriving, using the processing device, at least one label for each nodeusing a deep belief network, and identifying, using the processingdevice, a predetermined pattern in the graph based at least in part onthe labeled nodes.

Signal Analysis. Digital Signal Processing (DSP) is the use of digitalprocessing, such as by computers, to perform a wide variety of signalprocessing operations. The signals processed in this manner are asequence of numbers that represent samples of a continuous variable in adomain such as time, space, or frequency. Digital signal processing andanalog signal processing are subfields of signal processing. DSPapplications include audio and speech signal processing, sonar, radarand other sensor array processing, spectral estimation, statisticalsignal processing, digital image processing, signal processing fortelecommunications, control of systems, biomedical engineering, seismicdata processing, among others. Digital signal processing can involvelinear or nonlinear operations. Nonlinear signal processing is closelyrelated to nonlinear system identification and can be implemented in thetime, frequency, and spatio-temporal domains.

In DSP, digital signals are analyzed in one of the following domains:time domain (one-dimensional signals), spatial domain (multidimensionalsignals), frequency domain, and wavelet domains. The domain in which toprocess a signal is determined by making an informed assumption (or bytrying different possibilities) as to which domain best represents theessential characteristics of the signal. A sequence of samples from ameasuring device produces a temporal or spatial domain representation,whereas a discrete Fourier transform produces the frequency domaininformation, that is, the frequency spectrum. Signal analysis is furtherdescribed in Agilent Technologies Application Note 243 published 2000[5952-8898E) entitled: “The Fundamentals of Signal Processing”, which isincorporated in its entirety for all purposes as if fully set forthherein.

Time domain is the analysis of mathematical functions, physical signalsor time series of economic or environmental data, with respect to time.In the time domain, the signal or function's value is known for all realnumbers, for the case of continuous time, or at various separateinstants in the case of discrete time. An oscilloscope is a toolcommonly used to visualize real-world signals in the time domain. Atime-domain graph shows how a signal changes with time, whereas afrequency-domain graph shows how much of the signal lies within eachgiven frequency band over a range of frequencies.

In frequency domain analysis, also known as spectrum- or spectralanalysis, Signals are converted from time or space domain to thefrequency domain usually through the Fourier transform. The Fouriertransform converts the signal information to a magnitude and phasecomponent of each frequency. Often the Fourier transform is converted tothe power spectrum, which is the magnitude of each frequency componentsquared. The most common purpose for analysis of signals in thefrequency domain is analysis of signal properties. The engineer canstudy the spectrum to determine which frequencies are present in theinput signal and which are missing. There are some commonly usedfrequency domain transformations. For example, the cepstrum converts asignal to the frequency domain through Fourier transform, takes thelogarithm, then applies another Fourier transform. This emphasizes theharmonic structure of the original spectrum. Fourier Transform isdescribed in Lectures Notes entitled: “EE 261—The Fourier Transform andits Applications” by Prof. Brad Osgood of the Electrical EngineeringDepartment, Stanford University, downloaded from the Internet onNovember 2016, which is incorporated in its entirety for all purposes asif fully set forth herein.

A spectrum analyzer measures the magnitude of an input signal versusfrequency within the full frequency range of the instrument. The primaryuse is to measure the power of the spectrum of known and unknownsignals. The input signal that a spectrum analyzer measures iselectrical; however, spectral compositions of other signals, such asacoustic pressure waves and optical light waves, can be consideredthrough the use of an appropriate transducer. By analyzing the spectraof electrical signals, dominant frequency, power, distortion, harmonics,bandwidth, and other spectral components of a signal can be observedthat are not easily detectable in time domain waveforms. Theseparameters are useful in the characterization of electronic devices,such as wireless transmitters.

Spectrum analyzer types are distinguished by the methods used to obtainthe spectrum of a signal. There are swept-tuned and Fast FourierTransform (FFT) based spectrum analyzers. A swept-tuned analyzer uses asuperheterodyne receiver to down-convert a portion of the input signalspectrum to the center frequency of a narrow band-pass filter, whoseinstantaneous output power is recorded or displayed as a function oftime. By sweeping the receiver's center-frequency (using avoltage-controlled oscillator) through a range of frequencies, theoutput is also a function of frequency. While the sweep centers on anyparticular frequency, it may be missing short-duration events at otherfrequencies. An FFT analyzer computes a time-sequence of periodograms.FFT refers to a particular mathematical algorithm used in the process.This is commonly used in conjunction with a receiver andanalog-to-digital converter. As above, the receiver reduces thecenter-frequency of a portion of the input signal spectrum, but theportion is not swept. The purpose of the receiver is to reduce thesampling rate that is contended by the analyzer. With a sufficiently lowsample-rate, FFT analyzers can process all the samples (100%duty-cycle), and are therefore able to avoid missing short-durationevents. Spectrum analyzer basics are described in Agilent TechnologiesApplication Note 150 published Feb. 25, 2014 [5952-0292] entitled:“Spectrum Analysis Basics”, which is incorporated in its entirety forall purposes as if fully set forth herein.

Audio signal processing. Audio signal processing, sometimes referred toas audio processing, is the intentional alteration of auditory signals,or sound, often through an audio effect or effects unit. As audiosignals may be electronically represented in either digital or analogformat, signal processing may occur in either domain. Analog processorsoperate directly on the electrical signal, while digital processorsoperate mathematically on the digital representation of that signal.Audio signal processing is described in a book published 2003 by DavidRocchesso (Universita di Verona) entitled: “Introduction to SoundProcessing” [ISBN 88-901126-1-1], and in a book by Udo Zolzer of theTechnical University of Hamburg-Harburg, Germany published 1995 by JohnWiley & Sons, Ltd. [ISBN 0-47197226-6] entitled: “Digital audio SignalProcessing”, which are both incorporated in their entirety for allpurposes as if fully set forth herein. An example of a digital audioprocessor is IC model TDA7590 available from STMicroelectronics NV,described in a data sheet Rev. 3 entitled: “TDA7590—Digital signalprocessing IC for speech and audio applications” published 2013 bySTMicroelectronics, which is incorporated in its entirety for allpurposes as if fully set forth herein. Another example of DSP IC isModel No. TMS320C6678 available from Texas Instruments Incorporated,headquartered in Dallas, Tex., U.S.A., described in a Texas Instrumentsdata sheet SPRS691E—November 2010—Revised March 2014 entitled:“TMS320C6678—Multicore Fixed and Floating-Point Digital SignalProcessor”, which is incorporated in its entirety for all purposes as iffully set forth herein.

Audio signal processing typically involve analyzing, detecting,processing, simulating, or cancelling of the following affects orphenomena, or using the following techniques:

a. Echo: to simulate the effect of reverberation in a large hall orcavern, one or several delayed signals are added to the original signal.To be perceived as echo, the delay has to be of order 35 milliseconds orabove. Short of actually playing a sound in the desired environment, theeffect of echo can be implemented using either digital or analogmethods. Analog echo effects are implemented using tape delays and/orspring reverbs. When large numbers of delayed signals are mixed overseveral seconds, the resulting sound has the effect of being presentedin a large room, and it is more commonly called reverberation or reverbfor short.b. Flanger—to create an unusual sound, a delayed signal is added to theoriginal signal with a continuously variable delay (usually smaller than10 ms). This effect is now done electronically using a DSP, butoriginally the effect was created by playing the same recording on twosynchronized tape players, and then mixing the signals together. As longas the machines were synchronized, the mix would sound more-or-lessnormal, but if the operator placed his finger on the flange of one ofthe players (hence “flanger”), that machine would slow down and itssignal would fall out-of-phase with its partner, producing a phasingeffect. Once the operator took his finger off, the player would speed upuntil its tachometer was back in phase with the master, and as thishappened, the phasing effect would appear to slide up the frequencyspectrum. This phasing up-and-down the register can be performedrhythmically.c. Phaser—another way of creating an unusual sound; the signal is split,a portion is filtered with an all-pass filter to produce a phase-shift,and then the unfiltered and filtered signals are mixed. The phasereffect was originally a simpler implementation of the flanger effectsince delays were difficult to implement with analog equipment. Phasersare often used to give a “synthesized” or electronic effect to naturalsounds, such as human speech. The voice of C-3PO from Star Wars wascreated by taking the actor's voice and treating it with a phaser.d. Chorus—a delayed signal is added to the original signal with aconstant delay. The delay has to be short in order not to be perceivedas echo, but above 5 ms to be audible. If the delay is too short, itwill destructively interfere with the un-delayed signal and create aflanging effect. Often, the delayed signals will be slightly pitchshifted to more realistically convey the effect of multiple voices.e. Equalization—different frequency bands are attenuated or boosted toproduce desired spectral characteristics. Moderate use of equalization(often abbreviated as “EQ”) can be used to “fine-tune” the tone qualityof a recording; extreme use of equalization, such as heavily cutting acertain frequency can create more effects that are unusual.f. Filtering—Equalization is a form of filtering. In the general sense,frequency ranges can be emphasized or attenuated using low-pass,high-pass, band-pass or band-stop filters. Band-pass filtering of voicecan simulate the effect of a telephone because telephones use band-passfilters. overdrive effects such as the use of a fuzz box can be used toproduce distorted sounds, such as for imitating robotic voices or tosimulate distorted radiotelephone traffic. The most basic overdriveeffect involves clipping the signal when its absolute value exceeds acertain threshold.g. Pitch shift—this effect shifts a signal up or down in pitch. Forexample, a signal may be shifted an octave up or down. This is usuallyapplied to the entire signal and not to each note separately. Blendingthe original signal with shifted duplicate(s) can create harmonies fromone voice. Another application of pitch shifting is pitch correction.Here a musical signal is tuned to the correct pitch using digital signalprocessing techniques. This effect is ubiquitous in karaoke machines andis often used to assist pop singers who sing out of tune.h. Time stretching—the complement of pitch shift, that is, the processof changing the speed of an audio signal without affecting its pitch.i. Resonators—emphasize harmonic frequency content on specifiedfrequencies. These may be created from parametric EQs or fromdelay-based comb-filters.j. Synthesizer—generate artificially almost any sound by eitherimitating natural sounds or creating completely new sounds.k. Modulation—to change the frequency or amplitude of a carrier signalin relation to a predefined signal. Ring modulation, also known asamplitude modulation, is an effect made famous by Doctor Who's Daleksand commonly used throughout sci-fi.l. Compression—the reduction of the dynamic range of a sound to avoidunintentional fluctuation in the dynamics. Level compression is not tobe confused with audio data compression, where the amount of data isreduced without affecting the amplitude of the sound it represents.m. Reverse echo—a swelling effect created by reversing an audio signaland recording echo and/or delay while the signal runs in reverse. Whenplayed back forward the last echoes are heard before the effected soundcreating a rush like swell preceding and during playback.n. Active noise control—a method for reducing unwanted sound.

Mel-Frequency Cepstrum (MFC). The Mel-Frequency Cepstrum (MFC) is arepresentation of the short-term power spectrum of a sound, based on alinear cosine transform of a log power spectrum on a nonlinear Mel scaleof frequency. Mel-Frequency Cepstral Coefficients (MFCCs) arecoefficients that collectively make up an MFC. The difference betweenthe Cepstrum and the Mel-frequency Cepstrum is that in the MFC, thefrequency bands are equally spaced on the Mel scale, which approximatesthe human auditory system's response more closely than thelinearly-spaced frequency bands used in the normal Cepstrum. Thisfrequency warping can allow for better representation of sound, forexample, in audio compression.

MFCCs are commonly derived by the steps of: taking the Fourier transformof (a windowed excerpt of) a signal, mapping the powers of the spectrumobtained above onto the Mel scale, using triangular overlapping windows;taking the logs of the powers at each of the Mel frequencies; taking thediscrete cosine transform of the list of Mel log powers, as if it were asignal, and the MFCCs are the amplitudes of the resulting spectrum.There can be variations on this process, for example: differences in theshape or spacing of the windows used to map the scale, or addition ofdynamics features such as “delta” and “delta-delta” (first- andsecond-order frame-to-frame difference) coefficients. Calculating andusing MFCC is further described in European Telecommunications StandardsInstitute (ETSI) 2003 Standard ETSI ES 201 108 v1.1.3 (September 2003)entitled: “Speech Processing, Transmission and quality Aspects (STQ);Distributed speech recognition; Front-end feature extraction algorithm;Compression algorithms”, in an article in J. Computer Science &Technology, 16(6):582-589, September 2001 by Fang Zheng, Guoliang Zhang,and Zhanjiang Song entitled: “Comparison of Different Implementations ofMFCC”, and in RWTH Aachen, University of Technology, Aachen Germanypublication by Sirko Molau, Michael Pitz, Ralf Schluter, and HermannNey, entitled: “Computing MEL-Frequency Cepstral Coefficients on thePower Spectrum”, which are all incorporated in their entirety for allpurposes as if fully set forth herein.

LPC. Linear Predictive Coding (LPC) is a tool used mostly in audiosignal processing and speech processing for representing the spectralenvelope of a digital signal of speech in compressed form, using theinformation of a linear predictive model. It is one of the most powerfulspeech analysis techniques, and one of the most useful methods forencoding good quality speech at a low bit rate and provides extremelyaccurate estimates of speech parameters. LPC is described in aTechnion—Haifa, Israel presentation by Nimrod Peleg (Updated March 2009)entitled: “Linear Prediction Coding”, and in a book by P. P.Vaidyanathan entitled: “The Theory of Linear Prediction” published 2008[ISBN: 1598295756], which are both incorporated in their entirety forall purposes as if fully set forth herein.

LPC starts with the assumption that a speech signal is produced by abuzzer at the end of a tube (voiced sounds), with occasional addedhissing and popping sounds (sibilants and plosive sounds). Althoughapparently crude, this model is actually a close approximation of thereality of speech production. The glottis (the space between the vocalfolds) produces the buzz, which is characterized by its intensity(loudness) and frequency (pitch). The vocal tract (the throat and mouth)forms the tube, which is characterized by its resonances, which giverise to formants, or enhanced frequency bands in the sound produced.Hisses and pops are generated by the action of the tongue, lips andthroat during sibilants and plosives. LPC analyzes the speech signal byestimating the formants, removing their effects from the speech signal,and estimating the intensity and frequency of the remaining buzz. Theprocess of removing the formants is called inverse filtering, and theremaining signal after the subtraction of the filtered modeled signal iscalled the residue.

The numbers that describe the intensity and frequency of the buzz, theformants, and the residue signal, can be stored or transmitted somewhereelse. LPC synthesizes the speech signal by reversing the process: usethe buzz parameters and the residue to create a source signal, use theformants to create a filter (which represents the tube), and run thesource through the filter, resulting in speech. Because speech signalsvary with time, this process is done on short chunks of the speechsignal, which are called frames, generally 30 to 50 frames per secondgive intelligible speech with good compression. LPC is frequently usedfor transmitting spectral envelope information, and as thus has to betolerant of transmission errors. Transmission of the filter coefficientsdirectly is undesirable, since they are very sensitive to errors. Inother words, a very small error can distort the whole spectrum, orworse, a small error might make the prediction filter unstable. Thereare more advanced representations such as Log Area Ratios (LAR), LineSpectral Pairs (LSP) decomposition and reflection coefficients. Ofthese, especially LSP decomposition has gained popularity, since itensures stability of the predictor, and spectral errors are local forsmall coefficient deviations.

Time—frequency Analysis. A time-frequency analysis comprises thosetechniques that study a signal in both the time and frequency domainssimultaneously, using various time-frequency representations. Ratherthan viewing a 1-dimensional signal (a function, real or complex-valued,whose domain is the real line) and some transform (another functionwhose domain is the real line, obtained from the original via sometransform), time-frequency analysis studies a two-dimensional signal—afunction whose domain is the two-dimensional real plane, obtained fromthe signal via a time-frequency transform. Time-Frequency analysis isdescribed in an article by Rolf Hut (September 2004) entitled: “TimeFrequency Analysis —a Comparison between cochlear modeling and existingmethods”, and in an article by Franz Hlawatsch and Gerald Matz (of theInstitute of Communications and radio-Frequency Engineering, ViennaUniversity of Technology) entitled: “Time-Frequency Signal Processing: AStatistical Perspective”, which are both incorporated in their entiretyfor all purposes as if fully set forth herein. One of the most basicforms of time-frequency analysis is the Short-Time Fourier Transform(STFT), but techniques that are more sophisticated have been developed,such as wavelets.

There are several different ways to formulate a valid time-frequencydistribution function, resulting in several well-known time-frequencydistributions, such as: Short-time Fourier transform (including theGabor transform); Wavelet transform; Bilinear time-frequencydistribution function (Wigner distribution function, or WDF); andModified Wigner distribution function or Gabor-Wigner distributionfunction.

To analyze the signals well, choosing an appropriate time-frequencydistribution function is important. Which time-frequency distributionfunction should be used depends on the application being considered, asshown by reviewing a list of applications. The high clarity of theWigner Distribution Function (WDF) obtained for some signals is due tothe auto-correlation function inherent in its formulation; however, thelatter also causes the cross-term problem. Therefore, if we want toanalyze a single-term signal, using the WDF may be the best approach; ifthe signal is composed of multiple components, some other methods likethe Gabor transform, Gabor-Wigner distribution or ModifiedB-Distribution functions may be better choices.

HMD. A Head-Mounted Display (or Helmet-Mounted Display, for aviationapplications), both abbreviated HMD, is a display device, worn on thehead or as part of a helmet, that has a small display optic in front ofone (monocular HMD) or each eye (binocular HMD). There is also anOptical head-mounted display (OHMD), which is a wearable display thathas the capability of reflecting projected images as well as allowingthe user to see through it. A typical HMD has either one or two smalldisplays with lenses and semi-transparent mirrors embedded in a helmet,eyeglasses (also known as data glasses) or visor. The display units areminiaturized and may include CRT, LCDs, Liquid crystal on silicon(LCos), or OLED. Some vendors employ multiple micro-displays to increasetotal resolution and field of view. An HMD 47 b is pictorially depictedin FIG. 4 b , and includes a horizontal strap 48 a and a vertical strap48 b for head wearing by a person. A wireless-capable HMD 47 a ispictorially depicted in FIG. 4 b , shown to include an antenna 49 a andan antenna 49 b for wireless communication. The wireless-capable HMD 47a is shown worn by a person 36 in a view 47 c shown in FIG. 4 c.

HMDs differ in whether they can display just a Computer Generated Image(CGI), show live images from the real world or a combination of both.Most HMDs display only a computer-generated image, sometimes referred toas a virtual image. Some HMDs allow a CGI to be superimposed on areal-world view. This is sometimes referred to as augmented reality ormixed reality. Combining real-world view with CGI can be done byprojecting the CGI through a partially reflective mirror and viewing thereal world directly. This method is often called Optical See-Through.Combining real-world view with CGI can also be done electronically byaccepting video from a camera and mixing it electronically with CGI.This method is often called Video See-Through.

An optical head-mounted display uses an optical mixer, which is made ofpartly silvered mirrors. It has the capability of reflecting artificialimages as well as letting real images to cross the lens and let the userto look through it. Various techniques have existed for see-throughHMD's. Most of these techniques can be summarized into two mainfamilies: “Curved Mirror” based and “Waveguide” based. Various waveguidetechniques have existed for some time. These techniques includediffraction optics, holographic optics, polarized optics, and reflectiveoptics. Major HMD applications include military, governmental (fire,police, etc.) and civilian/commercial (medicine, video gaming, sports,etc.).

The Virtual Reality (VR) technology most fundamental to the proposedresearch is the Head-Mounted Display (HMD). An HMD is a helmet or visorworn by the user with two screens, one for each eye, so that astereoscopic “true 3D” image may be displayed to the user. This isachieved by displaying the same image in each screen, but offset by adistance equal to the distance between the user's eyes, mimicking howhuman vision perceives the world. HMDs can be opaque or see-through. Ina see-through HMD, the screens are transparent so that the user can seethe real world as well as what is being displayed on the screens.However, see-through HMDs often suffer from brightness problems thatmake them difficult to use in variable lighting conditions. Most opaqueHMD designs block out the real world so that the user can only see thescreens, thereby providing an immersive experience.

Some HMDs are used in conjunction with tracking systems. By tracking theuser's position or orientation (or both), the system can allow the userto move naturally via locomotion and by turning their head and body, andupdate the graphical display accordingly. This allows for naturalexploration of virtual environments without needing to rely on akeyboard, mouse, joystick, and similar interface hardware. Positionaltracking is often accomplished by attaching markers (such as infraredmarkers) to the HMD or the user's body and using multiple specialcameras to track the location of these markers in 3D space. Orientationtracking can be accomplished using an inertial tracker, which uses asensor to detect velocities on three axes. Some systems use acombination of optical and inertial tracking, and other trackingtechniques (e.g., magnetic) also exist. The output from the trackingsystems is fed into the computer rendering the graphical display so thatit can update the scene. Filtering is usually necessary to make the datausable since it comes in the form of noisy analog measurements. An HMDtypically includes a horizontal strap and a vertical strap for headwearing by a person. A wireless-capable HMD typically includes anantenna for wireless communication.

Methods and systems for capturing an image are provided in U.S. PatentApplication Publication No. 2013/0222638 to Wheeler et al. entitled:“Image Capture Based on Gaze Detection”, which is incorporated in itsentirety for all purposes as if fully set forth herein. In one example,a head-mounted device (HMD) having an image capturing device, aviewfinder, a gaze acquisition system, and a controller may beconfigured to capture an image. The image capturing device may beconfigured to have an imaging field of view including at least a portionof a field of view provided by the viewfinder. The gaze acquisitionsystem may be configured to acquire a gaze direction of a wearer. Thecontroller may be configured to determine whether the acquired gazedirection is through the viewfinder and generate an image captureinstruction based on a determination that the acquired gaze directionindicates a gaze through the viewfinder. The controller may further beconfigured to cause the image capturing device to capture an image.

Methods and systems for capturing and storing an image are provided inU.S. Pat. No. 8,941,561 to Starner entitled: “Image Capture”, which isincorporated in its entirety for all purposes as if fully set forthherein. In one example, eye-movement data associated with ahead-mountable device (HMD) may be received. The HMD may include animage-capture device arranged to capture image data corresponding to awearer-view associated with the HMD. In one case, the receivedeye-movement data may indicate sustained gaze. In this case, a locationof the sustained gaze may be determined, and an image including a viewof the location of the sustained gaze may be captured. At least oneindication of a context of the captured image, such as time and/orgeographic location of the HMD when the image was captured may bedetermined and stored in a data-item attribute database as part of arecord of the captured image. In a further example, movements associatedwith the HMD may also be determined and based on to determine sustainedgaze and the location of the sustained gaze.

A head mountable display (HMD) system is disclosed in U.S. PatentApplication Publication No. 2014/0362446 to Bickerstaff et al. entitled:“Electronic Correction Based on Eye Tracking”, which is incorporated inits entirety for all purposes as if fully set forth herein. The headmountable display (HMD) system comprises an eye position detectorcomprising one or more cameras configured to detect the position of eachof the HMD user's eyes; a dominant eye detector configured to detect adominant eye of the HMD user; and an image generator configured togenerate images for display by the HMD in dependence upon the HMD user'seye positions, the image generator being configured to apply a greaterweight to the detected position of the dominant eye than to the detectedposition of the non-dominant eye.

Methods and systems are described that involve a headmountable display(HMD) or an associated device determining the orientation of a person'shead relative to their body, are described in U.S. Pat. No. 9,268,136 toPatrick et al. entitled: “Use of Comparative Sensor Data to DetermineOrientation of Head Relative to Body”, which is incorporated in itsentirety for all purposes as if fully set forth herein. To do so,example methods and systems may compare sensor data from the HMD tocorresponding sensor data from a tracking device that is expected tomove in a manner that follows the wearer's body, such a mobile phonethat is located in the HMD wearer's pocket.

A Head Mountable Display (HMD) system in which images are generated fordisplay to the user is described in Patent Cooperation Treaty (PCT)International Application (IA) Publication No. WO 2014/199155 toAshforth et al. entitled: “Head-Mountable Apparatus and Systems”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The head mountable display (HMD) system comprises a detectorconfigured to detect the eye position and/or orientation and/or the headorientation of the HMD wearer, and a controller configured to controlthe generation of images for display, at least in part, according to thedetection of the eye position and/or orientation and/or the headorientation of the HMD wearer; in which the controller is configured tochange the display of one or more image features according to whether ornot the user is currently looking at those image features, the imagefeatures are menu items or information items, by rendering an imagefeature so as to be more prominent on the display if the user is lookingat it, such that the image feature is enlarged, moved from a peripheraldisplay position, replaced by a larger image feature and/or broughtforward in a 3D display space if the user is looking at it.

AR. Augmented reality (AR) is an interactive experience of a real-worldenvironment where the objects that reside in the real world are enhancedby computer-generated perceptual information, sometimes across multiplesensory modalities, including visual, auditory, haptic, somatosensoryand olfactory. AR can be defined as a system that fulfills three basicfeatures: a combination of real and virtual worlds, real-timeinteraction, and accurate 3D registration of virtual and real objects.The overlaid sensory information can be constructive (i.e., additive tothe natural environment), or destructive (i.e. masking of the naturalenvironment). This experience is seamlessly interwoven with the physicalworld such that it is perceived as an immersive aspect of the realenvironment. In this way, augmented reality alters one's ongoingperception of a real-world environment, whereas virtual realitycompletely replaces the user's real-world environment with a simulatedone. Augmented reality is related to two largely synonymous terms: mixedreality and computer-mediated reality.

The primary value of augmented reality is the manner in which componentsof the digital world blend into a person's perception of the real world,not as a simple display of data, but through the integration ofimmersive sensations, which are perceived as natural parts of anenvironment. Augmented reality is used to enhance natural environmentsor situations and offer perceptually enriched experiences. With the helpof advanced AR technologies (e.g. adding computer vision, incorporatingAR cameras into smartphone applications and object recognition) theinformation about the surrounding real world of the user becomesinteractive and digitally manipulated. Information about the environmentand its objects is overlaid on the real world. This information can bevirtual or real, such as seeing other real sensed or measuredinformation such as electromagnetic radio waves overlaid in exactalignment with where they actually are in space. Augmented reality alsohas a lot of potential in the gathering and sharing of tacit knowledge.Augmentation techniques are typically performed in real time and insemantic contexts with environmental elements. Immersive perceptualinformation is sometimes combined with supplemental information likescores over a live video feed of a sporting event. This combines thebenefits of both augmented reality technology and heads up displaytechnology (HUD).

Typical hardware components for augmented reality are: a processor,display, sensors and input devices. Modern mobile computing devices likesmartphones and tablet computers contain these elements, which ofteninclude a camera and microelectromechanical systems (MEMS) sensors suchas an accelerometer, GPS, and solid state compass, making them suitableAR platforms. There are two technologies used in augmented reality:diffractive waveguides and reflective waveguides.

Various technologies are used in augmented reality rendering, includingoptical projection systems, monitors, handheld devices, and displaysystems, which are worn on the human body. A Head-Mounted Display (HMD)is a display device worn on the forehead, such as a harness orhelmet-mounted. HMDs place images of both the physical world and virtualobjects over the user's field of view. Modern HMDs often employ sensorsfor six degrees of freedom monitoring that allow the system to alignvirtual information to the physical world and adjust accordingly withthe user's head movements. HMDs can provide VR users with mobile andcollaborative experiences.

AR displays can be rendered on devices resembling eyeglasses. Versionsinclude eyewear that employs cameras to intercept the real world viewand re-display its augmented view through the eyepieces and devices inwhich the AR imagery is projected through or reflected off the surfacesof the eyewear lens pieces.

HUD. A Head-Up Display (HUD) is a transparent display that presents datawithout requiring users to look away from their usual viewpoints.Near-eye augmented reality devices can be used as portable head-updisplays as they can show data, information, and images while the userviews the real world. Many definitions of augmented reality only defineit as overlaying the information. This is basically what a head-updisplay does; however, practically speaking, augmented reality isexpected to include registration and tracking between the superimposedperceptions, sensations, information, data, and images and some portionof the real world.

Contact lenses. Contact lenses that display AR imaging are indevelopment. These bionic contact lenses might contain the elements fordisplay embedded into the lens including integrated circuitry, LEDs andan antenna for wireless communication.

Virtual retinal display. A Virtual Retinal Display (VRD) is a personaldisplay device where a display is scanned directly onto the retina of aviewer's eye. This results in bright images with high resolution andhigh contrast, and the viewer sees what appears to be a conventionaldisplay floating in space. Virtual retinal display creates images thatcan be seen in ambient daylight and ambient room light. The VRD isconsidered a preferred candidate to use in a surgical display due to itscombination of high resolution and high contrast and brightness.Additional tests show high potential for VRD to be used as a displaytechnology for patients that have low vision.

Handheld. A Handheld display employs a small display that fits in auser's hand. All handheld AR solutions to date opt for videosee-through. Initially handheld AR employed fiducial markers, and laterGPS units and MEMS sensors such as digital compasses and six degrees offreedom accelerometer-gyroscope. Today Simultaneous Localization andMapping (SLAM) markerless trackers such as PTAM (Parallel Tracking andMapping) are starting to come into use. Handheld display AR promises tobe the first commercial success for AR technologies. The two mainadvantages of handheld AR are the portable nature of handheld devicesand the ubiquitous nature of camera phones. The disadvantages are thephysical constraints of the user having to hold the handheld device outin front of them at all times, as well as the distorting effect ofclassically wide-angled mobile phone cameras when compared to the realworld as viewed through the eye.

Spatial Augmented Reality (SAR) augments real-world objects and scenes,without the use of special displays such as monitors, head-mounteddisplays or hand-held devices. SAR makes use of digital projectors todisplay graphical information onto physical objects. The key differencein SAR is that the display is separated from the users of the system.Since the displays are not associated with each user, SAR scalesnaturally up to groups of users, allowing for collocated collaborationbetween users.

Other applications include table and wall projections. One innovation,the Extended Virtual Table, separates the virtual from the real byincluding beam-splitter mirrors attached to the ceiling at an adjustableangle. Virtual showcases, which employ beam splitter mirrors togetherwith multiple graphics displays, provide an interactive means ofsimultaneously engaging with the virtual and the real. Many moreimplementations and configurations make spatial augmented realitydisplay an increasingly attractive interactive alternative. A SAR systemcan display on any number of surfaces in an indoor setting at once. SARsupports both a graphical visualization and passive haptic sensation forthe end users. Users are able to touch physical objects in a processthat provides passive haptic sensation.

A virtual reality composer platform and system (VRCPS) is described inU.S. Pat. No. 7,754,955 to Egan entitled: “Virtual reality composerplatform system”, which is incorporated in its entirety for all purposesas if fully set forth herein. The system includes a plurality of userinput/output devices and signal input/output controllers interfaced to acentral processing unit complete with plurality of memory means, abutterfly morpheus musical instrument with plurality of manual inputmeans each with a first unique visible indicia interfaced to saidcentral processing unit, a plurality of finger adapters each with asecond unique visible indicia donned on respective fingers and at leastone custom butterfly Morpheus music notation computer interface. Thesystem is particularly suited for composing music for self-learning andteaching for all types of musical instruments for optimal virtualreality multimedia experience. The VRCPS platform and concepts disclosedare vari-dimensional acoustic environments, which are equally suited toall types of electronic learning and composing systems, game systems andcomputers. It is suitable for all levels of Do-It-Yourself learning fromlearning Beginners to Virtuoso Levels.

A method, apparatus, and User Interface, and product for assisting userslearning to play the Chords of any selected Song are described in U.S.Pat. No. 10,614,786 to Barry entitled: “Musical chord identification,selection and playing method and means for physical and virtual musicalinstruments”, which is incorporated in its entirety for all purposes asif fully set forth herein. The method, apparatus, and User Interface,and product quickly and easily provide means to quickly and easilygenerate the individual Note sounds for the Chords of the selected Songemploying a broad range of Virtual and Physical Instrument.

An augmented reality based piano performance assistant method whichenables a user who is not familiar with a sheet music to play a piano,and a device performing the same is provided in South-Korea PatentApplication Publication KR20170138135 entitled: “Method of helping pianoperformance and based on augmented reality and apparatus performing thesame”, which is incorporated in its entirety for all purposes as iffully set forth herein. According to an embodiment of the presentinvention, the augmented reality based piano performance assistantmethod which is performed by an augmented reality based pianoperformance assistant device comprises the following steps of:recognizing a plurality of octave recognition labels located on akeyboard of the piano device; loading a virtual keyboard with respect tokeyboard information in an augmented reality environment by using theinformation on the octave corresponding to the plurality of octaverecognition labels; and displaying piano performance assistantinformation on the virtual keyboard loaded in the augmented realityenvironment according to performance information received from aperformance information providing server when performance is started.

A kind of piano training system and method based on mixed reality isdescribed in China Patent Application Publication CN109493686 entitled:“A kind of piano training system and method based on mixed reality”,which is incorporated in its entirety for all purposes as if fully setforth herein. The piano training system includes the mixed realityhelmet, piano, positioning device and processing unit; the piano istrue. The positioning device is arranged on the piano or the mixedreality helmet, for relative position between the key and the mixedreality helmet of the real-time measurement piano. The processing unit,signal connect the mixed reality helmet, for loading and according tothe corresponding virtual training scene of standard piano filegenerated, and by the virtual training scene transfer to the mixedreality helmet. The mixed reality helmet, signal connects thepositioning device, for user's body-worn, the virtual training scene isshown in the upside of the key of the piano for relative positionbetween the key and the key and the mixed reality helmet of the pianothat detect according to the positioning device of the real piano. Itsenjoyment that can increase piano training process and permission useradjust the sitting posture of oneself in the training process to preventcervical vertebra over fatigue.

A computer implemented method for providing an augmented reality (AR)function is described in U.S. Pat. No. 10,482,862 to Hämäläinen et al.entitled: “Computer implemented method for providing augmented reality(AR) function regarding music track”, which is incorporated in itsentirety for all purposes as if fully set forth herein. The methodcomprises receiving input information regarding a music track and aninstrument; determining attribute information of the music track basedon the received input information; receiving real time content ofaudiovisual (AV) input signals using at least one capturing device;generating visual information corresponding to visual data of the realtime content, wherein the visual information corresponds to a viewregarding at least one user limb and an instrument comprising aplurality of user operable elements; generating augmented reality (AR)instruction information based on the attribute information of the musictrack, the augmented reality (AR) instruction information comprising aplurality of layers; and generating augmented reality (AR) visualinformation by applying the augmented reality (AR) instructioninformation to the visual information so that a first layer of theaugmented reality (AR) instruction information is applied above at leasta portion of the visual information.

A system enabling the performance of sensory stimulating contentincluding music and video using gaming in a cyber reality environment,such as using a virtual reality headset, is described in U.S. Pat. No.10,418,008 to Bencar et al. entitled: “Cyber reality device includinggaming based on a plurality of musical programs”, which is incorporatedin its entirety for all purposes as if fully set forth herein. Thisdisclosure includes a system and method through which a performer canvirtually trigger and control a presentation of pre-packaged sensorystimulating content including musical programs through gaming. A themefor the performer is that the pre-packaged sensory stimulating contentis preferably chosen such that, even where the performer is a novice,the sensory stimulating data is presented in a pleasing and sympatheticmanner and scoring is provided as a function of the performer's abilityto provide a gesture in association with a displayed virtual trigger.

Musical notation. Music notation (or musical notation) is any system,convention, or standard used to visually represent aurally perceivedmusic played with musical instruments or sung by the human voice throughthe use of written, printed, or otherwise-produced symbols, includingnotation for durations of absence of sound such as rests. Typically, inthis framework pitches are indicated by placing oval noteheads on thestaff lines or between the lines, and the pitch of the oval musicalnoteheads can be modified by accidentals. The duration (note length) isshown with different note values, which can be indicated by the noteheadbeing a stemless hollow oval (a whole note or semibreve), a hollowrectangle or stemless hollow oval with one or two vertical lines oneither side (double whole note or breve), a stemmed hollow oval (a halfnote or minim), or solid oval using stems to indicate quarter notes(crotchets) and stems with added flags or beams to indicate smallersubdivisions, and additional symbols such as dots and ties whichlengthen the duration of a note. Notation is read from left to right,which makes setting music for right-to-left scripts difficult.

A staff (or stave, in British English) of written music generally beginswith a clef, which indicates the position of one particular note on thestaff. The treble clef or G clef was originally a letter G and itidentifies the second line up on the five line staff as the note G abovemiddle C. The bass clef or F clef shows the position of the note F belowmiddle C. While the treble and bass clef are the most widely used clefs,other clefs are used, such as the alto clef (used for viola and altotrombone music) and the tenor clef (used for some cello, tenor trombone,and double bass music). Notes representing a pitch outside of the scopeof the five line staff can be represented using ledger lines, whichprovide a single note with additional lines and spaces. Some instrumentsuse mainly one clef, such as violin and flute, which use treble clef anddouble bass and tuba, which use bass clef. Some instruments regularlyuse both clefs, such as piano and pipe organ.

Following the clef, the key signature on a staff indicates the key ofthe piece or song by specifying that certain notes are flat or sharpthroughout the piece, unless otherwise indicated with accidentals addedbefore certain notes. When a sharp is placed before a note, this makesthat note one semitone higher. When a flat is placed before a note, thismakes that note one semitone lower. Double sharps and double flats areless common, but they are used. A double sharp is placed before a noteto make it two semitones higher. A double flat is placed before a noteto make it two semitones lower. A natural sign placed before a noterenders that note in its “natural” form, which means that any sharps orflats applying to that note from the key signature or from accidentalsare cancelled. Sometimes a courtesy accidental is used in music where itis not technically required, to remind the musician of what pitch thekey signature requires. Following the key signature is the timesignature. The time signature typically consists of two numbers, withone of the most common being The top “4” indicates that there are fourbeats per measure (also called bar). The bottom “4” indicates that eachof those beats are quarter notes. Measures divide the piece into groupsof beats, and the time signatures specify those groupings.

Musical symbols. Musical symbols are marks and symbols used in musicalnotation of musical scores. Some are used to notate pitch, tempo, metre,duration, and articulation of a note or a passage of music. In somecases, symbols provide information about the form of a piece (e.g., howmany repeats of a section) or about how to play the note (e.g., withviolin family instruments, a note may be bowed or plucked). Some symbolsare instrument-specific notation giving the performer information aboutwhich finger, hand or foot to use. Selected examples of popular musicalsymbols according to a popular music notation convention are describedin a view 60 shown in FIG. 6 , and a correspondence of musical symbolsto the associated piano keys is described in a view 60 a shown in FIG. 6a.

Clefs define the pitch range, or tessitura, of the staff on which it isplaced. A clef is usually the leftmost symbol on a staff. Additionalclefs may appear in the middle of a staff to indicate a change inregister for instruments with a wide range. In early music, clefs couldbe placed on any of several lines on a staff. Musical note and restvalues are not absolutely defined, but are proportional in duration toall other note and rest values. The whole note is the reference value,and the other notes are named (in American usage) in comparison; i.e., aquarter note is a quarter of the length of a whole note. Accidentalsmodify the pitch of the notes that follow them on the same staffposition within a measure, unless cancelled by an additional accidental.Key signatures define the prevailing key of the music that follows, thusavoiding the use of accidentals for many notes. If no key signatureappears, the key is assumed to be C major/A minor, but can also signifya neutral key, employing individual accidentals as required for eachnote. The key signature examples shown here are described as they wouldappear on a treble staff.

Time signatures define the meter of the music. Music is “marked off” inuniform sections called bars or measures, and time signatures establishthe number of beats in each. This does not necessarily indicate whichbeats to emphasize, however, so a time signature that conveysinformation about the way the piece actually sounds is thus chosen. Timesignatures tend to suggest prevailing groupings of beats or pulses.Articulations (or accents) specify how to perform individual noteswithin a phrase or passage. They can be fine-tuned by combining morethan one such symbol over or under a note. They may also appear inconjunction with phrasing marks listed above.

Sheet music. Sheet music is a handwritten or printed form of musicalnotation that uses musical symbols to indicate the pitches, rhythms, orchords of a song or instrumental musical piece. The medium of sheetmusic traditionally was a paper, however modern mediums include thepresentation of musical notation on computer screens and the developmentof scorewriter computer programs that can notate a song or pieceelectronically, and, in some cases, “play back” the notated music usinga synthesizer or virtual instruments.

Sheet music can be used as a record of, a guide to, or a means toperform, a song or a musical piece. Sheet music enables instrumentalperformers who are able to read music notation (a pianist, orchestralinstrument players, a jazz band, etc.) or singers to perform a song orpiece. The intended purpose of an edition of sheet music affects itsdesign and layout. If sheet music is intended for study purposes, as ina music history class, the notes and staff can be made smaller and theeditor does not have to be worried about page turns. In classical music,authoritative musical information about a piece can be gained bystudying the written sketches and early versions of compositions thatthe composer might have retained, as well as the final autograph scoreand personal markings on proofs and printed scores. An example of asheet music of a popular song is described in a view 60 b shown in FIG.6 b.

Musical instrument. A musical instrument is a device created or adaptedto make musical sounds. There are many different methods of classifyingmusical instruments. Various methods examine aspects such as thephysical properties of the instrument (material, color, shape, etc.),the use for the instrument, the means by which music is produced withthe instrument, the range of the instrument, and the instrument's placein an orchestra or other ensemble. Most methods are specific to ageographic area or cultural group and were developed to serve the uniqueclassification requirements of the group.

The most commonly used system divides instruments into stringinstruments, woodwind instruments, brass instruments and percussioninstruments. Musical instruments are also often classified by theirmusical range in comparison with other instruments in the same family.This may be useful when placing instruments in context of an orchestraor other ensemble. These terms are named after singing voiceclassifications, and include Soprano instruments, such as flute, violin,soprano saxophone, trumpet, clarinet, oboe, and piccolo; Altoinstruments, such as alto saxophone, French horn, English horn, viola,and alto horn; Tenor instruments, such as trombone, tenoroon, tenorsaxophone, tenor violin, guitar, and tenor drum; Baritone instruments,such as bassoon, baritone saxophone, bass clarinet, cello, baritonehorn, and euphonium; and Bass instruments, such as double bass, bassguitar, contrabassoon, bass saxophone, tuba, and bass drum.

Some instruments fall into more than one category. For example, thecello may be considered tenor, baritone or bass, depending on how itsmusic fits into the ensemble. The trombone and French horn may be alto,tenor, baritone, or bass depending on the range it is played in. Manyinstruments have their range as part of their name: soprano saxophone,tenor saxophone, baritone horn, alto flute, bass guitar, etc. Additionaladjectives describe instruments above the soprano range or below thebass, for example the sopranino saxophone and contrabass clarinet. Whenused in the name of an instrument, these terms are relative, describingthe instrument's range in comparison to other instruments of its familyand not in comparison to the human voice range or instruments of otherfamilies.

The original Hornbostel-Sachs system classified instruments into fourmain groups: Idiophones, which produce sound by vibrating the primarybody of the instrument itself; they are sorted into concussion,percussion, shaken, scraped, split, and plucked idiophones, such asclaves, xylophone guiro, slit drum, mbira, and rattle; Membranophones,which produce sound by a vibrating a stretched membrane; they may bedrums (further sorted by the shape of the shell), which are struck byhand, with a stick, or rubbed, but kazoos and other instruments that usea stretched membrane for the primary sound (not simply to modify soundproduced in another way) are also considered membranophones;Chordophones, which produce sound by vibrating one or more strings; theyare sorted into according to the relationship between the string(s) andthe sounding board or chamber (for example, if the strings are laid outparallel to the sounding board and there is no neck, the instrument is azither whether it is plucked like an autoharp or struck with hammerslike a piano. If the instrument has strings parallel to the soundingboard or chamber and the strings extend past the board with a neck, thenthe instrument is a lute, whether the sound chamber is constructed ofwood like a guitar or uses a membrane like a banjo); Aerophones, whichproduce a sound with a vibrating column of air; they are sorted intofree aerophones such as a bullroarer or whip, which move freely throughthe air; reedless aerophones, such as flutes and recorders, which causethe air to pass over a sharp edge; reed instruments, which use avibrating reed (this category may be further divided into twoclassifications: single-reeded and double-reeded instruments. Examplesof the former are clarinets and saxophones, while the latter includesoboes and bassoons); and lip-vibrated aerophones such as trumpets,trombones and tubas, for which the lips themselves function as vibratingreeds.

String instruments. String instruments, also known as stringedinstruments, or chordophones, are musical instruments that produce soundfrom vibrating strings when the performer plays or sounds the strings insome manner. Musicians play some string instruments by plucking thestrings with their fingers or a plectrum—and others by hitting thestrings with a light wooden hammer or by rubbing the strings with a bow.In some keyboard instruments, such as the harpsichord, the musicianpresses a key that plucks the string. With bowed instruments, the playerpulls a rosined horsehair bow across the strings, causing them tovibrate. With a hurdy-gurdy, the musician cranks a wheel whose rosinededge touches the strings. In most string instruments, the vibrations aretransmitted to the body of the instrument, which often incorporates somesort of hollow or enclosed area. The body of the instrument alsovibrates, along with the air inside it. The vibration of the body of theinstrument and the enclosed hollow or chamber make the vibration of thestring more audible to the performer and audience. The body of moststring instruments is hollow. Some, however—such as electric guitar andother instruments that rely on electronic amplification—may have a solidwood body. The most common string instruments in the string family areguitar, electric bass, violin, viola, cello, double bass, banjo,mandolin, ukulele, and harp. Pictorial views of a violin 51, a guitar 51a, a mandolin 51 b, a banjo 51 c, and a harp 51 d, are shown in FIG. 5 .

Bowed instruments include the string section instruments of theClassical music orchestra (violin, viola, cello and double bass) and anumber of other instruments (e.g., viols and gambas used in early musicfrom the Baroque music era and fiddles used in many types of folkmusic). All of the bowed string instruments can also be plucked with thefingers, a technique called “pizzicato”. A wide variety of techniquesare used to sound notes on the electric guitar, including plucking withthe fingernails or a plectrum, strumming and even “tapping” on thefingerboard and using feedback from a loud, distorted guitar amplifierto produce a sustained sound. Some types of string instrument are mainlyplucked, such as the harp and the electric bass. Other examples includethe sitar, rebab, banjo, mandolin, ukulele, and bouzouki.

String instruments can be divided in three groups: Lutes, which areinstruments that support the strings via a neck and a bout (“gourd”),for instance a guitar, a violin, or a saz, Harps, which are instrumentsthat contain the strings within a frame, and Zithers, which areinstruments that have the strings mounted on a body, frame or tube, suchas a guqin, a cimbalom, an autoharp, harpsichord, a piano, or a valiha.

It is also possible to divide the instruments into categories focused onhow the instrument is played. An acoustic guitar being strummed. Allstring instruments produce sound from one or more vibrating strings,transferred to the air by the body of the instrument (or by a pickup inthe case of electronically amplified instruments). They are usuallycategorised by the technique used to make the strings vibrate (or by theprimary technique, in the case of instruments where more than one mayapply). The three most common techniques are plucking, bowing, andstriking. An important difference between bowing and plucking is that inthe former the phenomenon is periodic so that the overtones are kept ina strictly harmonic relationship to the fundamental.

Plucking is a method of playing on instruments such as the veena, banjo,ukulele, guitar, harp, lute, mandolin, oud, and sitar, using either afinger, thumb, or quills (now plastic plectra) to pluck the strings.Instruments normally played by bowing may also be plucked, a techniquereferred to by the Italian term pizzicato.

Bowing is a method used in some string instruments, including theviolin, viola, cello, and the double bass (of the violin family), andthe old viol family. The bow consists of a stick with a “ribbon” ofparallel horse tail hairs stretched between its ends. The hair is coatedwith rosin so it can grip the string; moving the hair across a stringcauses a stick-slip phenomenon, making the string vibrate, and promptingthe instrument to emit sound. Darker grades of rosin grip well in cool,dry climates, but may be too sticky in warmer, more humid weather.Violin and viola players generally use harder, lighter-colored rosinthan players of lower-pitched instruments, who tend to favor darker,softer rosin.

The third common method of sound production in stringed instruments isto strike the string. The piano and hammered dulcimer use this method ofsound production. Even though the piano strikes the strings, the use offelt hammers means that the sound that is produced can nevertheless bemellow and rounded, in contrast to the sharp attack produced when a veryhard hammer strikes the strings. Violin family string instrument playersare occasionally instructed to strike the string with the stick of thebow, a technique called col legno. This yields a percussive sound alongwith the pitch of the note.

Acoustic instruments are based on a vibrating string strung on a verythick log, as a hypothetical example, would make only a very quietsound, so string instruments are usually constructed in such a way thatthe vibrating string is coupled to a hollow resonating chamber, asoundboard, or both. On the violin, for example, the four strings passover a thin wooden bridge resting on a hollow box (the body of theviolin). The normal force applied to the body from the strings issupported in part by a small cylinder of wood called the soundpost. Theviolin body also has two “f-holes” carved on the top. The strings'vibrations are distributed via the bridge and soundpost to all surfacesof the instrument, and are thus made louder by matching of the acousticimpedance. The correct technical explanation is that they allow a bettermatch to the acoustic impedance of the air.

All lute type instruments traditionally have a bridge, which holds thestring at the proper action height from the fret/finger board at one endof the strings. On acoustic instruments, the bridge performs an equallyimportant function of transmitting string energy into the “sound box” ofthe instrument, thereby increasing the sound volume. The specificdesign, and materials the used in the construction of the bridge of aninstrument, have a dramatic impact upon both the sound andresponsiveness of the instrument. Acoustic instruments can also be madeout of artificial materials, such as carbon fiber and fiberglass(particularly the larger, lower-pitched instruments, such as cellos andbasses).

String instruments may be based on electronic amplification, where theyare fitted with piezoelectric or magnetic pickups to convert thestring's vibrations into an electrical signal that is amplified and thenconverted back into sound by loudspeakers. Some players attach a pickupto their traditional string instrument to “electrify” it. Another optionis to use a solid-bodied instrument, which reduces unwanted feedbackhowls or squeals. Amplified string instruments can be much louder thantheir acoustic counterparts, so musicians can play them in relativelyloud rock, blues, and jazz ensembles. Amplified instruments can alsohave their amplified tone modified by using electronic effects such asdistortion, reverb, or wah-wah. Bass-register string instruments such asthe double bass and the electric bass are amplified with bass instrumentamplifiers that are designed to reproduce low-frequency sounds. Tomodify the tone of amplified bass instruments, a range of electronicbass effects are available, such as distortion and chorus.

Woodwind instrument. A woodwind instrument is a musical instrument whichproduces sound when the player blows air against a sharp edge or througha reed, causing the air within its resonator (usually a column of air)to vibrate. Most of these instruments are made of wood but can be madeof other materials, such as metals or plastics. Woodwind instruments area family of musical instruments within the more general category of windinstruments. Common examples include flute, clarinet, oboe, saxophone,and bassoon. There are two main types of woodwind instruments: flutesand reed instruments (otherwise called reed pipes). The main distinctionbetween these instruments and other wind instruments is the way in whichthey produce sound. All woodwinds produce sound by splitting the airblown into them on a sharp edge, such as a reed or a fipple. Despite thename, a woodwind may be made of any material, not just wood, and commonexamples include brass, silver, cane, as well as other metals such asgold and platinum. The saxophone, for example, though made of brass, isconsidered a woodwind because it requires a reed to produce sound.

The modern orchestra's woodwind section typically includes: flutes,oboes, clarinets, and bassoons. The piccolo, cor anglais, bass clarinet,E-flat clarinet, and contrabassoon are commonly used supplementarywoodwind instruments. The section may also on occasion be expanded bythe addition of saxophone(s). The concert band's woodwind sectiontypically includes piccolos, flutes, oboes, bass clarinets, bassoons,alto saxophones, tenor saxophones, and baritone saxophones. Pictorialviews of a Flute 52, a Piccolo 52 a, a Clarinet 52 b, a Bass Clarinet 52c, a Bassoon 52 d, a Contra Bassoon 52 e, an Oboe 52 f, and an EnglishHorn 52 g are shown in FIG. 5 a . Woodwind instruments are typicallydivided into 2 groups: flutes and reed instruments.

Flutes produce sound by directing a focused stream of air below the edgeof a hole in a cylindrical tube. The flute family can be divided intotwo sub-families: open flutes and closed flutes. To produce a sound withan open flute, the player is required to blow a stream of air across asharp edge that then splits the airstream. This split airstream thenacts upon the air column contained within the flute's hollow causing itto vibrate and produce sound. To produce a sound with a closed flute,the player is required to blow air into a duct. This duct acts as achannel bringing the air to a sharp edge. As with the open flutes, theair is then split; this causes the column of air within the closed fluteto vibrate and produce sound. Examples of this type of flute include therecorder, ocarina, and organ pipes.

Reed instruments produce sound by focusing air into a mouthpiece whichthen causes a reed, or reeds, to vibrate. Similarly to flutes, reedpipes are also further divided into two types: single reed and doublereed. Single-reed woodwinds produce sound by fixing a reed onto theopening of a mouthpiece (using a ligature). When air is forced betweenthe reed and the mouthpiece, the reed causes the air column in theinstrument to vibrate and produce its unique sound. Single reedinstruments include the clarinet, saxophone, and others such as thechalumeau. Double-reed instruments use two precisely cut, small piecesof cane bound together at the base. This form of sound production hasbeen estimated to have originated in the middle to late Neolithicperiod; its discovery has been attributed to the observation of windblowing through a split rush. The finished, bound reed is inserted intothe instrument and vibrates as air is forced between the two pieces(again, causing the air within the instrument to vibrate as well). Thisfamily of reed pipes is subdivided further into another twosub-families: exposed double reed, and capped double reed instruments.Exposed double-reed instruments are played by having the double reeddirectly between the player's lips. This family includes instrumentssuch as the oboe, cor anglais (also called English horn) and bassoon,and many types of shawms throughout the world. On the other hand, cappeddouble-reed instruments have the double reed covered by a cap. Theplayer blows through a hole in this cap that then directs the airthrough the reeds. This family includes the crumhorn.

Brass instrument. A brass instrument is a musical instrument thatproduces sound by sympathetic vibration of air in a tubular resonator insympathy with the vibration of the player's lips. There are severalfactors involved in producing different pitches on a brass instrument.Slides, valves, crooks (though they are rarely used today), or keys areused to change vibratory length of tubing, thus changing the availableharmonic series, while the player's embouchure, lip tension and air flowserve to select the specific harmonic produced from the availableseries.

Because the player of a brass instrument has direct control of the primevibrator (the lips), brass instruments exploit the player's ability toselect the harmonic at which the instrument's column of air vibrates. Bymaking the instrument about twice as long as the equivalent woodwindinstrument and starting with the second harmonic, players can get a goodrange of notes simply by varying the tension of their lips. Most brassinstruments are fitted with a removable mouthpiece. Different shapes,sizes and styles of mouthpiece may be used to suit differentembouchures, or to more easily produce certain tonal characteristics.Trumpets, trombones, and tubas are characteristically fitted with acupped mouthpiece, while horns are fitted with a conical mouthpiece. Oneinteresting difference between a woodwind instrument and a brassinstrument is that woodwind instruments are non-directional. This meansthat the sound produced propagates in all directions with approximatelyequal volume. Brass instruments, on the other hand, are highlydirectional, with most of the sound produced traveling straight outwardfrom the bell. This difference makes it significantly more difficult torecord a brass instrument accurately. It also plays a major role in someperformance situations, such as in marching bands. Pictorial views of aTrumpet 53, a Tuba 53 a, a French Horn 53 b, and a Cornet 53 d are shownin FIG. 5 b.

Modern brass instruments generally come in one of two families: Valvedbrass instruments use a set of valves (typically three or four but asmany as seven or more in some cases) operated by the player's fingersthat introduce additional tubing, or crooks, into the instrument,changing its overall length. This family includes all of the modernbrass instruments except the trombone: the trumpet, horn (also calledFrench horn), euphonium, and tuba, as well as the cornet, flugelhorn,tenor horn (alto horn), baritone horn, sousaphone, and the mellophone.As valved instruments are predominant among the brasses today, a morethorough discussion of their workings can be found below. The valves areusually piston valves, but can be rotary valves; the latter are the normfor the horn (except in France) and are also common on the tuba.

Slide brass instruments use a slide to change the length of tubing. Themain instruments in this category are the trombone family, though valvetrombones are occasionally used, especially in jazz. The trombonefamily's ancestor, the sackbut, and the folk instrument bazooka are alsoin the slide family. There are two other families that have, in general,become functionally obsolete for practical purposes. Instruments of bothtypes, however, are sometimes used for period-instrument performances ofBaroque or Classical pieces. In more modern compositions, they areoccasionally used for their intonation or tone color.

Percussion instrument. A percussion instrument is a musical instrumentthat is sounded by being struck or scraped by a beater includingattached or enclosed beaters or rattles struck, scraped or rubbed byhand or struck against another similar instrument. The percussionsection of an orchestra most commonly contains instruments such as thetimpani, snare drum, bass drum, cymbals, triangle and tambourine, aswell as keyboard percussion instruments such as the glockenspiel andxylophone. However, the section can also contain non-percussiveinstruments, such as whistles and sirens, or a blown conch shell.Percussive techniques can even be applied to the human body itself, asin body percussion. Percussion instruments are most commonly dividedinto two classes: Pitched percussion instruments, which produce noteswith an identifiable pitch, and unpitched percussion instruments, whichproduce notes or sounds in an indefinite pitch. A pictorial view of acommon drum set 54 is shown in FIG. 5 c , which includes a Floor torn 54a, a Snare drum 54 b, Tom-toms 54 c, a Hi-hat 54 d, crash and ridecymbals 54 e, and a bass drum 54 f. Percussion instruments may play notonly rhythm, but also melody and harmony. Because of the diversity ofpercussive instruments, it is not uncommon to find large musicalensembles composed entirely of percussion. Rhythm, melody, and harmonyare all represented in these ensembles. Percussion instruments areclassified by various criteria sometimes depending on theirconstruction, ethnic origin, function within musical theory andorchestration, or their relative prevalence in common knowledge.

Piano. The piano is an acoustic, stringed musical instrument, in whichthe strings are struck by wooden hammers that are coated with a softermaterial (modern hammers are covered with dense wool felt; some earlypianos used leather). It is played using a keyboard, which is a row ofkeys (small levers) that the performer presses down or strikes with thefingers and thumbs of both hands to cause the hammers to strike thestrings. A piano usually has a protective wooden case surrounding thesoundboard and metal strings, which are strung under great tension on aheavy metal frame. Pressing one or more keys on the piano's keyboardcauses a wooden or plastic hammer (typically padded with firm felt) tostrike the strings. The hammer rebounds from the strings, and thestrings continue to vibrate at their resonant frequency. Thesevibrations are transmitted through a bridge to a soundboard thatamplifies by more efficiently coupling the acoustic energy to the air.When the key is released, a damper stops the strings' vibration, endingthe sound. Notes can be sustained, even when the keys are released bythe fingers and thumbs, by the use of pedals at the base of theinstrument. The sustain pedal enables pianists to play musical passagesthat would otherwise be impossible, such as sounding a 10-note chord inthe lower register and then, while this chord is being continued withthe sustain pedal, shifting both hands to the treble range to play amelody and arpeggios over the top of this sustained chord. Unlike thepipe organ and harpsichord, two major keyboard instruments widely usedbefore the piano, the piano allows gradations of volume and toneaccording to how forcefully or softly a performer presses or strikes thekeys.

Most modern pianos have a row of 88 black and white keys, 52 white keysfor the notes of the C major scale (C, D, E, F, G, A and B) and 36shorter black keys, which are raised above the white keys, and setfurther back on the keyboard. This means that the piano can play 88different pitches (or “notes”), going from the deepest bass range to thehighest treble. The black keys are for the “accidentals”, which areneeded to play in all twelve keys. More rarely, some pianos haveadditional keys (which require additional strings). Most notes havethree strings, except for the bass, which graduates from one to two. Thestrings are sounded when keys are pressed or struck, and silenced bydampers when the hands are lifted from the keyboard. There are threeprimary types of pianos: Grand, Upright, and Electronic.

Grands are the largest piano type, and frequently the most majestic (aswell as expensive). Grand pianos are characterized by horizontalsoundboards, which sometimes stretch up to 4 ft. (front to back). Thesoundboard is encased in a supportable opening platform that liftsupwards on the right. Dampers lie on top of the strings, adjacent to thehammers (also horizontal). The internal construction is braced withform-holders, usually made of wood, as well as the small equipped metalreinforcements. The casing is essentially “bottomless”, allowing one tosee the soundboard support base, also of reinforced wood, whichtechnically acts as the base. Keys consist of wood coated in ivory, orsometimes pure ivory, depending on the piano's manufacturers andclassification. The grand piano has the standard 88 keys. Most of thesepianos have sheet music stands. A retractable cover slides over, orfolds down on the keys.

The baby grand piano, as the name implies, is essentially a smallerversion of the regular grand piano. It has an 88-key set up, just likethe regular grand does, but generally has a smaller soundboard and thusis not as loud as the regular grand.

Uprights are the most common type of acoustic piano and are a popularaddition to a living room or parlor. The upright piano has been afavorite because it costs less, is more compact, and offers a warmsound. The soundboard is vertical, with strings that stretch downwardand horizontal hammers and dampers. The hammers strike horizontally andare returned to resting position by springs, taking slightly longer thana grand piano's hammers (which are vertical and returned by the force ofgravity). The support base of the soundboard is visible on the backside,as well as wooden reinforcements. Uprights usually cost less, dependingon the model; however, some can exceed grands in total value. Althoughuprights often get depicted as inferior to the grand pianos, a five-footupright can rival a typical grand in terms of tone quality and loudness.Like the grand, upright pianos vary in material construction.

Suitable for beginners or moving performers, electric pianos are usuallythe most affordable, and although they do not have the qualities of anacoustic, sound continues to improve for the high-end and mid-rangeinstruments. They vary greatly in quality, some have hollow keys, whileothers try to replicate the feel and weight of acoustic keyboards. Inaddition to the features of an acoustic piano, electric pianos have avariety of sounds and settings such as organ, guitar, string, choir, andpercussion. The numerous sounds on some keyboards make it virtually aportable band. Other pianos have limited functions, but this is betterfor someone who is trying to replicate an acoustic and save money. Trueelectric pianos (compared to the plain keyboards) have a professionalappearance and good materials (most consist largely of plastic), as wellas touch-sensitive features and sometimes equipped frames. Most haveconnectors for pedals and computer interactive abilities. They neverneed to be tuned, and rapidly becoming more popular in modern bands. Theelectric piano also has the advantage of allowing the user to practicesilently with headphones at times when doing so would otherwise disturbpeople. The few drawbacks include technological infancy and therequirement of a power supply.

Tempo. A ‘tempo’ is the speed or pace of a given piece, and is typicallyindicated with an instruction at the start of a piece (often usingconventional Italian terms) and is usually measured in beats per minute(or bpm). In modern classical compositions, a “metronome mark” in beatsper minute may supplement or replace the normal tempo marking, while inmodern genres like electronic dance music, tempo will typically simplybe stated in bpm. While the ability to hold a steady tempo is a vitalskill for a musical performer, tempo is changeable. Depending on thegenre of a piece of music and the performers' interpretation, a piecemay be played with slight tempo rubato or drastic variances. Inensembles, the tempo is often indicated by a conductor or by one of theinstrumentalists, for instance the drummer. While tempo is described orindicated in many different ways, including with a range of words (e.g.,“Slowly”, “Adagio” and so on), it is typically measured in beats perminute (bpm or BPM). For example, a tempo of 60 beats per minutesignifies one beat per second, while a tempo of 120 beats per minute istwice as rapid, signifying one beat every 0.5 seconds. The note value ofa beat will typically be that indicated by the denominator of the timesignature.

Tempo is not necessarily fixed throughout a musical piece. Within apiece (or within a movement of a longer work), a composer may indicate acomplete change of tempo, often by using a double bar and introducing anew tempo indication, often with a new time signature and/or keysignature.

Examples of common tempo markings, ranging from slowest to fastest tempoare: Larghissimo—very, very slowly (24 bpm and under); Adagissimo—veryslowly; Grave—very slow (25-45 bpm), Largo —broadly (40-60 bpm); Lento—slowly (45-60 bpm); Larghetto —rather broadly (60-66 bpm); Adagio—slowly with great expression (66-76 bpm); Adagietto —slower thanandante (72-76 bpm) or slightly faster than adagio (70-80 bpm); Andante—at a walking pace (76-108 bpm); Andantino —slightly faster than andante(although, in some cases, it can be taken to mean slightly slower thanandante) (80-108 bpm); Marcia moderato —moderately, in the manner of amarch (83-85 bpm); Andante moderato —between andante and moderato (thusthe name) (92-98 bpm); Moderato —at a moderate speed (98-112 bpm);Allegretto—moderately fast (102-110 bpm); Allegro moderato —close to,but not quite allegro (116-120 bpm); Allegro —fast, quick, and bright(120-156 bpm) (molto allegro is slightly faster than allegro, but alwaysin its range of 124-156 bpm); Vivace—lively and fast (156-176 bpm);Vivacissimo—very fast and lively (172-176 bpm); Allegrissimo or Allegrovivace—very fast (172-176 bpm); Presto—very, very fast (168-200 bpm);and Prestissimo —even faster than presto (200 bpm and over).

MIDI. Musical Instrument Digital Interface (MIDI) is a technicalstandard that describes a communications protocol, digital interface,and electrical connectors that connect a wide variety of electronicmusical instruments, computers, and related audio devices for playing,editing and recording music. MIDI technology is standardized by the MIDIManufacturers Association (MMA) and the MIDI Committee of theAssociation of Musical Electronics Industry (AMEI) in Tokyo. Version 1.0of the MIDI standard is described in the standard entitled: “TheComplete MIDI 1.0 Detailed Specification Incorporating all RecommendedPractices document”, version 96.1 third edition, Published 2014 by TheMIDI Manufacturers Association Los Angeles, Calif., U.S.A., and Version2.0 of the standard is entitled: “MIDI 2.0 Specification Overview, MIDI2.0 Specification Overview MIDI Capability Inquiry (MIDI-CI) CommonRules for MIDI-CI Profiles, Common Rules for MIDI-CI Property Exchange,Universal MIDI Packet (UMP) Format, and MIDI 2.0 Protocol” Version 1.0Published Feb. 20, 2020 by Association of Musical Electronics IndustryAMEI and MIDI Manufacturers Association MMA, and an overview of MIDI isdescribed in a document entitled: “What is MIDI?” by Paul D. Lehrman,published 2017 by Tufts University, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

A single MIDI link through a MIDI cable can carry up to sixteen channelsof information, each of which can be routed to a separate device orinstrument. This could be sixteen different digital instruments, forexample. MIDI carries event messages; data that specify the instructionsfor music, including a note's notation, pitch, velocity (which is heardtypically as loudness or softness of volume); vibrato; panning to theright or left of stereo; and clock signals (which set tempo). When amusician plays a MIDI instrument, all of the key presses, buttonpresses, knob turns and slider changes are converted into MIDI data. Onecommon MIDI application is to play a MIDI keyboard or other controllerand use it to trigger a digital sound module (which contains synthesizedmusical sounds) to generate sounds, which the audience hears produced bya keyboard amplifier. MIDI data can be transferred via MIDI or USBcable, or recorded to a sequencer or digital audio workstation to beedited or played back.

MIDI file. The Standard MIDI File (SMF) is a file format that provides astandardized way for music sequences to be saved, transported, andopened in other systems. The standard was developed and is maintained bythe MMA, and usually uses a .mid extension. The compact size of thesefiles led to their widespread use in computers, mobile phone ringtones,webpage authoring and musical greeting cards. These files are intendedfor universal use and include such information as note values, timingand track names. Lyrics may be included as metadata, and can bedisplayed by karaoke machines. SMFs are created as an export format ofsoftware sequencers or hardware workstations. They organize MIDImessages into one or more parallel tracks and timestamp the events sothat they can be played back in sequence. A header contains thearrangement's track count, tempo and an indicator of which of three SMFformats the file uses. A type 0 file contains the entire performance,merged onto a single track, while type 1 files may contain any number oftracks that are performed in synchrony. Type 2 files are rarely used andstore multiple arrangements, with each arrangement having its own trackand intended to be played in sequence. Microsoft Windows bundles SMFstogether with Downloadable Sounds (DLS) in a Resource Interchange FileFormat (RIFF) wrapper, as RMID files with a .rmi extension. RIFF-RMIDhas been deprecated in favor of Extensible Music Files (XMF).

MIDI Controller. A MIDI controller is any hardware or software thatgenerates and transmits Musical Instrument Digital Interface (MIDI) datato MIDI-enabled devices, typically to trigger sounds and controlparameters of an electronic music performance. MIDI controllerstypically have some type of interface which the performer presses,strikes, blows or touches. This action generates MIDI data (e.g., notesplayed and their intensity), which can then be transmitted to aMIDI-compatible sound module or synthesizer using a MIDI cable. Thesound module or synthesizer in turn produces a sound which is amplifiedthrough a loudspeaker. The most commonly used MIDI controller is theelectronic musical keyboard MIDI controller. When the keys are played,the MIDI controller sends MIDI data about the pitch of the note, howhard the note was played and its duration. Other common MIDI controllersare wind controllers, which a musician blows into and presses keys totransmit MIDI data, and electronic drums. The MIDI controller can bepopulated with any number of sliders, knobs, buttons, pedals and othersensors, and may or may not include a piano keyboard.

MIDI Keyboard. A MIDI keyboard or controller keyboard is typically apiano-style electronic musical keyboard, often with other buttons,wheels and sliders, used for sending MIDI signals or commands over a USBor MIDI 5-pin cable to other musical devices or computers. MIDIkeyboards lacking an onboard sound module cannot produce soundsthemselves, however some models of MIDI keyboards contain both a MIDIcontroller and sound module, allowing them to operate independently.When used as a MIDI controller, MIDI information on keys or buttons theperformer has pressed is sent to a receiving device capable of creatingsound through modeling synthesis, sample playback, or an analog hardwareinstrument.

Musical piece. A musical piece, also referred to as musical composition,music composition, or simply composition, typically refers to anoriginal piece or work of music, either vocal or instrumental, thestructure of a musical piece, or to the process of creating or writing anew piece of music. People who create new compositions are calledcomposers. Composers of primarily songs are usually called songwriters;with songs, the person who writes lyrics for a song is the lyricist. Ingeneral, composition consists in two things—The first is the orderingand disposing of several sounds in such a manner that their successionpleases the ear, and the second is the rendering audible of two or moresimultaneous sounds in such a manner that their combination is pleasant.

The most popular music genres are classical music and popular music.While in popular music, chords are much more predictable and repetitivethan in classical music. For example, a typical Bach piece uses dozensof different chords in amazingly unique combinations. Further, populartunes generally use a lot of repeated notes, short repeated melodicphrases, and very simple melodic lines. Songs are easy to learn, easy toremember, and simple and fun to sing or play. Classical melodies have amuch more complex structure, tend to have longer repeated phrases, andcan be much more challenging, and more rewarding, to learn and perform.The main advantage popular music has over classical music is that popmusic tends to be more rhythmically sophisticated. Drum patterns in popmusic layer different rhythms for bass drum, snare drum, and hi hat.Most classical music is typically not as rhythmically interesting. Forinstance, some Bach pieces consist of nothing but straight sixteenthnotes.

Popular music. Popular music is music with wide appeal that is typicallydistributed to large audiences through the music industry. These formsand styles can be enjoyed and performed by people with little or nomusical training. Typically, popular music, unlike art music, isconceived for mass distribution to large and often socioculturallyheterogeneous groups of listeners, is stored and distributed innon-written form, is only possible in an industrial monetary economywhere it becomes a commodity and is commonly subject to commercialconsiderations. Popular music is found on most commercial and publicservice radio stations, in most commercial music retailers anddepartment stores, and movie and television soundtracks.

Common genres of popular music include ‘Rock music’ that originated as“rock and roll” in the United States in the early 1950s, and developedinto a range of different styles in the 1960s and later, particularly inthe United Kingdom and in the United States, Electronic music employselectronic musical instruments, digital instruments and circuitry-basedmusic technology, ‘Soul music’ that originated in the African Americancommunity throughout the United States in the 1950s and early 1960s andcombines elements of African-American gospel music, rhythm and blues andjazz, ‘Funk music’ that de-emphasizes melody and chord progressions andfocuses on a strong rhythmic groove of a bassline played by an electricbassist and a drum part played by a drummer, often at slower tempos thanother popular music, ‘Country music’ that often consists of ballads anddance tunes with generally simple forms, folk lyrics, and harmoniesmostly accompanied by string instruments such as banjos, electric andacoustic guitars, steel guitars (such as pedal steels and dobros), andfiddles as well as harmonicas, ‘Latin music’ that comes from Spanish-and Portuguese-speaking areas of the world, ‘Reggae music’ thatincorporates different stylistic techniques form rhythm and blues, jazz,African, Caribbean, and other genres as well but what makes reggaeunique are the vocals and lyrics, ‘Hip-hop music’ that is broadlydefined as a stylized rhythmic music that commonly accompanies rapping,a rhythmic and rhyming speech that is chanted, and ‘Polka music’ that isa dance music.

Classical music. Examples of classical music forms are Aria—which refersto a long accompanied song for a solo voice, typically one in an operaor oratorio, and is typically a formal musical composition that is aself-contained piece for one voice, with or without instrumental ororchestral accompaniment, normally part of a larger work; Cadenza—whichrefers to a virtuoso solo passage inserted into a movement in a concertoor other work, typically near the end, and generally includes animprovised or written-out ornamental passage played or sung by a soloistor soloists, usually in a “free” rhythmic style, and often allowingvirtuosic display, while during this time the accompaniment will rest,or sustain a note or chord; Concerto—which is a musical composition fora solo instrument or instruments accompanied by an orchestra, especiallyone conceived on a relatively large scale, and includes an instrumentalcomposition, written for one or more soloists accompanied by anorchestra or other ensemble. The typical three-movement structure, aslow movement (e.g., lento or adagio) preceded and followed by fastmovements (e.g. presto or allegro); Chamber music—which is aninstrumental music played by a small ensemble, with one player to apart, the most important form being the string quartet which developedin the 18th century, and typically is a form of classical music that iscomposed for a small group of instruments —traditionally a group thatcould fit in a palace chamber or a large room. Most broadly, it includesany art music that is performed by a small number of performers, withone performer to a part (in contrast to orchestral music, in which eachstring part is played by a number of performers); Movement—which is aprincipal division of a longer musical work, self-sufficient in terms ofkey, tempo, and structure, and while individual or selected movementsfrom a composition are sometimes performed separately, a performance ofthe complete work requires all the movements to be performed insuccession; Sonata—which is a composition for an instrumental soloist,often with a piano accompaniment, typically in several movements withone or more in sonata form; Opera —which is a dramatic work in one ormore acts, set to music for singers and instrumentalists, and where themusic has a leading role and the parts are taken by singers, but isdistinct from musical theatre, typically involving a collaborationbetween a composer and a librettist and incorporating a number of theperforming arts, such as acting, scenery, costume, and sometimes danceor ballet; and Overture—which is an orchestral piece at the beginning ofan opera, suite, play, oratorio, or other extended composition, andSymphony—which is an elaborate musical composition for full orchestra,typically in four movements, at least one of which is traditionally insonata form.

Vocal music. Vocal music is a type of singing performed by one or moresingers, either with instrumental accompaniment, or without instrumentalaccompaniment (a cappella), in which singing provides the main focus ofthe piece. Music which employs singing but does not feature itprominently is generally considered to be instrumental music as is musicwithout singing. Music without any non-vocal instrumental accompanimentis referred to as a cappella. Vocal music typically features sung wordscalled lyrics, although there are notable examples of vocal music thatare performed using non-linguistic syllables, sounds, or noises,sometimes as musical onomatopoeia, such as jazz scat singing. A shortpiece of vocal music with lyrics is broadly termed a song, although indifferent styles of music, it may be called an aria or hymn.

Vocal music often has a sequence of sustained pitches that rise andfall, creating a melody, but some vocal styles use less distinctpitches, such as chants or a rhythmic speech-like delivery, such asrapping. There are extended vocal techniques that may be used, such asscreaming, growling, throat singing, or yodelling.

Chroma features. Chroma features or chromagram, also referred to as“pitch class profiles”, relates to the twelve different pitch classes,and are a powerful tool for analyzing music whose pitches can bemeaningfully categorized (often into twelve categories) and whose tuningapproximates to the equal-tempered scale. One main property of chromafeatures is that they capture harmonic and melodic characteristics ofmusic, while being robust to changes in timbre and instrumentation. Theunderlying observation is that humans perceive two musical pitches assimilar in color if they differ by an octave. Based on this observation,a pitch can be separated into two components, which are referred to astone height and chroma. Assuming the equal-tempered scale, one considerstwelve chroma values represented by the set {C, C♯, D, D♯, E, F, F♯, G,G♯, A, A♯, B} that consists of the twelve pitch spelling attributes asused in Western music notation. Note that in the equal-tempered scaledifferent pitch spellings such C♯ and D♭ refer to the same chroma.Enumerating the chroma values, one can identify the set of chroma valueswith the set of integers {1, 2, . . . , 12}, where 1 refers to chroma C,2 to C♯, and so on. A pitch class is defined as the set of all pitchesthat share the same chroma. For example, using the scientific pitchnotation, the pitch class corresponding to the chroma C is the set { . .. , C−2, C−1, C0, C1, C2, C3 . . . } consisting of all pitches separatedby an integer number of octaves. Given a music representation (e.g. amusical score or an audio recording), the main idea of chroma featuresis to aggregate for a given local time window (e.g. specified in beatsor in seconds) all information that relates to a given chroma into asingle coefficient. Shifting the time window across the musicrepresentation results in a sequence of chroma features each expressinghow the representation's pitch content within the time window is spreadover the twelve chroma bands. The resulting time-chroma representationis also referred to as chromagram. The figure above shows chromagramsfor a C-major scale, once obtained from a musical score and once from anaudio recording. Because of the close relation between the terms chromaand pitch class, chroma features are also referred to as pitch classprofiles.

A chroma toolbox is presented in a paper entitled: “CHROMA TOOLBOX:MATLAB IMPLEMENTATIONS FOR EXTRACTING VARIANTS OF CHROMA-BASED AUDIOFEATURES” by Meinard Muller and Sebastian Ewert, published 2011 by theInternational Society for Music Information Retrieval, which isincorporated in its entirety for all purposes as if fully set forthherein. Chroma-based audio features, which closely correlate to theaspect of harmony, are a well-established tool in processing andanalyzing music data. There are many ways of computing and enhancingchroma features, which results in a large number of chroma variants withdifferent properties. The toolbox contains MATLAB implementations forextracting various types of recently proposed pitch-based andchroma-based audio features. Providing the MATLAB implementations on awell-documented website under a GNU-GPL license, our aim is to fosterresearch in music information retrieval. As another goal, we want toraise awareness that there is no single chroma variant that works bestin all applications. To this end, we discuss two example applicationsshowing that the final music analysis result may crucially depend on theinitial feature design step.

Music Arrangement. As used herein, the term “arrangement” in the contextof ‘music arrangement’ refers to any reworking of a musical piece (orpart thereof), that may be any composition or any original piece ofmusic. In one example, an arrangement may refer to adapted version of amusical piece, such as adaptation of the corresponding sheet music, sothat it can be played by a different instrument or combination ofinstruments from the original. For example, a song written for one voicewith piano accompaniment might be arranged so that it can be sung inparts by a choir, or a piece for violin might be arranged so that it canbe played on a clarinet instead. In another example, an arrangement mayrefer to adapted version of a musical piece, such as simplifying orelaboration of the corresponding sheet music, so that it can be playedby less skilled players or beginners, while retaining the generalcharacter of the original musical piece.

A statistical-modeling method for piano reduction, such as converting anensemble score into piano scores that can control performancedifficulty, is presented in an article entitled: “Statistical pianoreduction controlling performance difficulty” by Eita Nakamura andKazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published 2018 in SIP(2018), vol. 7, e13, which is incorporated in its entirety for allpurposes as if fully set forth herein. While previous studies havefocused on describing the condition for playable piano scores, itdepends on player's skill and can change continuously with the tempo.The article describes computationally quantifying performance difficultyas well as musical fidelity to the original score, and formulate theproblem as optimization of musical fidelity under constraints ondifficulty values. First, performance difficulty measures are developedby means of probabilistic generative models for piano scores and therelation to the rate of performance errors is studied. Second, todescribe musical fidelity, we construct a probabilistic modelintegrating a prior piano-score model and a model representing howensemble scores are likely to be edited. An iterative optimizationalgorithm for piano reduction is developed based on statisticalinference of the model. We confirm the effect of the iterativeprocedure; we find that subjective difficulty and musical fidelitymonotonically increase with controlled difficulty values; and we showthat incorporating sequential dependence of pitches and fingering motionin the piano-score model improves the quality of reduction scores inhigh-difficulty cases.

An automatic arrangement system for piano reduction that arranges musicalgorithmically for the piano while considering various roles of thepiano in music is presented in an article entitled: “Automatic Systemfor the Arrangement of Piano Reductions” by Shih-Chuan Chiu, Man-KwanShan, and Jiun-Long Huang, published 2009 in the 11th IEEE InternationalSymposium on Multimedia, which is incorporated in its entirety for allpurposes as if fully set forth herein. The automatic arrangement systemfor piano reduction is achieved by first analyzing the original music inorder to determine the type of arrangement element performed by aninstrument. Then each phrase is identified and is associated with aweighted importance value. At last, a phrase selection algorithm isproposed to select phrases with maximum importance to arrangement underthe constraint of piano playability. Our experiments demonstrate thatthe proposed system has the ability to create piano arrangement.

Looking for a piano sheet music with proper difficulty for a pianolearner is always an important work to his/her teacher. A study on a newand challenging issue of recognizing the difficulty level of piano sheetmusic is described in an article entitled: “A Study on Difficulty LevelRecognition of Piano Sheet Music” by Shih-Chuan Chiu and Min-Syan Chen,published December 2012 in the ISM '12: Proceedings of the 2012 IEEEInternational Symposium on Multimedia, which is incorporated in itsentirety for all purposes as if fully set forth herein. To analyze thesemantic content of music, we focus on symbolic music, i.e., sheet musicor score. Specifically, difficulty level recognition is formulated as aregression problem to predict the difficulty level of piano sheet music.Since the existing symbolic music features are not able to capture thecharacteristics of difficulty, we propose a set of new features. Toimprove the performance, a feature selection approach, RReliefF, is usedto select relevant features. An extensive performance study is conductedover two real datasets with different characteristics to evaluate theaccuracy of the regression approach for predicting difficulty level. Thebest performance evaluated in terms of the R2 statistics over twodatasets reaches 39.9% and 38.8%, respectively.

Huge sheet music collections exist on the Web, allowing people to accesspublic domain scores for free. However, beginners may be lost in findinga score appropriate to their instrument level, and should often rely onthemselves to start out on the chosen piece. In this instrumentale-Learning context, a Score Analyzer prototype in order to automaticallyextract the difficulty level of a MusicXML piece and suggest advicethanks to a Musical Sign Base (MSB) is proposed in an article entitled:“SCORE ANALYZER: AUTOMATICALLY DETERMINING SCORES DIFFICULTY LEVEL FORINSTRUMENTAL E-LEARNING” by Véronique Sébastien, Henri Ralambondrainy,Olivier Sébastien, and Noel Conruyt of IREMIA—Laboratoire d'Informatiqueet de Mathématiques, EA2525 University of Reunion Island, Saint-Denis,Reunion (FRANCE), published October 2012 in 13th International Societyfor Music Information Retrieval Conference (ISMIR 2012), which isincorporated in its entirety for all purposes as if fully set forthherein. The article first review methods related to score performanceinformation retrieval, and then identifies seven criteria tocharacterize technical instrumental difficulties and propose methods toextract them from a MusicXML score. The relevance of these criteria isthen evaluated through a Principal Components Analysis and compared tohuman estimations. Lastly, the article discusses the integration of thiswork to @-MUSE, a collaborative score annotation platform based onmultimedia contents indexation.

In the Western classical tradition, musicians play music from notatedsheet music, called a score. When playing music from a score, a musiciantranslates its visual symbols into sequences of instrument-specificphysical motions. Hence, a music score's overall complexity represents asum of the cognitive and mechanical acuity required for its performance.For a given instrument, different notes, intervals, articulations,dynamics, key signatures, and tempo represent dissimilar levels ofdifficulty, which vary depending on the performer's proficiency.Individual musicians embrace this tenet, but may disagree about thedegrees of difficulty. A ‘musiplectics’, a systematic and objectiveapproach to computational assessment of the complexity of a music scorefor any instrument, is introduced in a paper entitled: “Musiplectics:Computational Assessment of the Complexity of Music Scores” by EthanHolder, Eli Tilevich, and Amy Gillick, published October 2015 in ONWARD'15 [ACM 978-1-4503-1995-9/13/10,http://dx.doi.org/10.1145/2508075.2514879], which is incorporated in itsentirety for all purposes as if fully set forth herein. Musiplecticsdefines computing paradigms for automatically and accurately calculatingthe complexity of playing a music score on a given instrument. The coreconcept codifies a two-phase process. First, music experts rank therelative difficulty of individual musical components (e.g., notes,intervals, dynamics, etc.) for different playing proficiencies andinstruments. Second, a computing engine automatically applies thisranking to music scores and calculates their respective complexity. As aproof of concept of musiplectics, we present an automated, Web-basedapplication called Musical Complexity Scoring (MCS) for music educatorsand performers. Musiplectics can engender the creation of practicalcomputing tools for objective and expeditious assessment of a musicscore's suitability for the abilities of intended performers.

While the difficulty of the music can be classified by a variety ofstandard, conventional methods are classified by the subjective judgmentbased on the experience of many musicians or conductors. Music score isdifficult to evaluate as there is no quantitative criterion to determinethe degree of difficulty. A new classification method for determiningthe degree of difficulty of the music is proposed in a paper entitled:“A Method for Measuring the Difficulty of Music Scores” by Yang-Eui Songand Yong Kyu Lee [www.ksci.re.krhttp://dx.doi.org/10.9708/jksci.2016.21.4.039] published April 2016 inthe Journal of The Korea Society of Computer and Information Vol. 21 No.4, which is incorporated in its entirety for all purposes as if fullyset forth herein. In order to determine the degree of difficulty, weconvert the score, which is expressed as a traditional music score, intoelectronic music sheet. Moreover, we calculate information about theelements needed to play sheet music by distance of notes, tempo, andquantifying the ease of interpretation. Calculating a degree ofdifficulty of the entire music via the numerical data, we suggest thedifficulty evaluation of the score, and show the difficulty of musicthrough experiments.

Commercial application for writing music or songs are available, such asLudwig (http://www.write-music.com/), which is a music software forWindows that helps you to write your own songs, while you simply enteror play a melody. Ludwig does the rest: he finds the proper chords andwrites all parts of a professionally sounding band. It was never easierto compose, arrange or accompany songs, and AnthemScore(https://www.lunaverus.com/), which is the leading software forautomatic music transcription. Convert mp3, way, and other audio formatsinto sheet music using a neural network trained on millions of datasamples. Use powerful editing tools to tweak notes, beats, and timesignatures. Print or save as PDF, MIDI, or XML.

Online music learning. Increasing Internet penetration, smartphones andtablets in the modern and emerging economies are boosting the growth ofthe online music learning market, as an alternative to the traditionalpersonal one-on-one piano lessons. Various online courses and individuallessons that offer enhanced music skills are provided as applicationsfor mobile phones and tablets, typically based on Windows, iOS, Android,and MacOS platforms, are offered by multiple key players, supportingvarious musical instruments such as guitar, ukulele, piano, vocal,drums, and bass. In addition to the reduced costs, such online musiclearning provides an interactive and fun experience, while providingmore flexibility and being less bound by location and time, compared topersonal based tutoring. A pictorial illustration of an online musiclearning is described in a view 60 c in FIG. 6 c , showing a user 36that plays a piano 83 according to musical symbols visualized poon adisplay as part of a tablet 35 a.

Methods and apparatus that help users learn to play one or moreinstruments by employing simplified note recognition techniques,improved feedback, and a simplified keyboard tablature are provided inU.S. Pat. No. 7,030,307 to Wedel entitled: “Music teaching device andmethod”, which is incorporated in its entirety for all purposes as iffully set forth herein. In one aspect, users are encouraged to playspecified notes within a specified time frame. The pitch and durationcan be represented in any suitable manner, but are advantageouslydisplayed together as a single icon (232A-232F). Preferably, the iconhas an elongated shape in which the horizontal length correlates withduration of the note, and its vertical position on a display correlateswith a pitch of the note. Notes can thus be represented by elongatedlines, bars, ellipses, or even missiles or arrows. In another aspect, animproved musical tablature effectively clusters juxtaposed black keys(412A, 412C, 412E) and white keys (412B, 412D) on a display for easiervisualization.

A musical keyboard that is connected to a computer is described in U.S.Pat. No. 7,174,510 to Salter entitled: “Interactive game providinginstruction in musical notation and in learning an instrument”, which isincorporated in its entirety for all purposes as if fully set forthherein. The computer implements a graphical user interface for teachingusers to play the musical instrument. A computer readable music file,such as a MIDI file, is used to drive the creation of game objects thattravel from a point of origination along a path toward a key of avirtual keyboard. In one form, when a user presses a key of the musicalkeyboard within a certain time window of arrival of the game object atthe corresponding key of the virtual keyboard, the user is awarded withan audio presentation, a visual presentation and/or with game points. Ina more structured learning mode, the game can be played with selectable,progressively more difficult challenges that the user masters on theroad to proficiency.

A system and method for improving musical education through use of agame is disclosed in U.S. Pat. No. 9,492,756 to Izkovsky et al.entitled: “System and method for analyzing a digitalized musicalperformance”, which is incorporated in its entirety for all purposes asif fully set forth herein. The method includes the steps of: receivingelectrical signals associated with a musical piece provided by a user ofthe game; converting the electrical signals into digital samples; andanalyzing the digital samples with use of auxiliary information, forpurposes of improving signal analysis accuracy, resulting in determiningvarious parameters of the musical piece provided by the user, whereinthe auxiliary information is a-priori data related to at least oneelement selected from the group consisting of a musical instrument beingplayed, a technical environment, the game, and information regarding theuser of the game. Visual and/or audio feedback may also be provided tothe user regarding the musical piece that they are providing.

A system and method for providing exercise in playing a music instrumentis disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled:“System and method for providing exercise in playing a musicinstrument”, which is incorporated in its entirety for all purposes asif fully set forth herein. With prior art solutions there is a commonproblem of inadequate motivation of the user to continue practicing, andinadequate learning of items which are difficult for a specific user.The present solution detects characteristics of the user's play detectedand uses them to provide a suitable program of exercises and to providefeedback which enhances motivation of the user.

A system for providing a user a virtual exercise in playing a musicinstrument relative to the user's skill characteristics is disclosed inU.S. Pat. No. 9,767,705 to Klapuri et al. entitled: “System forestimating user's skill in playing a music instrument and determiningvirtual exercises thereof”, which is incorporated in its entirety forall purposes as if fully set forth herein. The system includes: aprocessing entity and a memory entity for processing and storing data,respectively to execute the system functions, and a data transfer entityfor receiving and transmitting data, the system configured to: obtainmusical notation data, analyze it to assign the musical piece to whichsuch data pertains a number of difficulty characteristics with scalarvalues, provide the user with a number of musical pieces, with knowndifficulty characteristics, as virtual exercises to be completed byplaying an instrument, obtain user performance data of completed virtualexercises, analyze the user performance data to determine and assign theuser with a number of skill characteristics values in accordance withthe difficulty characteristic values of the completed musical pieces,and determine a musical piece for the user as a virtual exercise.

A system and method for providing exercise in playing a music instrumentis disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled:“System and method for providing exercise in playing a musicinstrument”, which is incorporated in its entirety for all purposes asif fully set forth herein. With prior art solutions there is a commonproblem of inadequate motivation of the user to continue practicing, andinadequate learning of items which are difficult for a specific user.The present solution detects characteristics of the user's play detectedand uses them to provide a suitable program of exercises and to providefeedback which enhances motivation of the user.

Providing real-time interaction between a first player and a secondplayer to collaborate for a musical performance over a network isdisclosed in U.S. Pat. No. 10,182,093 to Klapuri entitled: “Computerimplemented method for providing real-time interaction between firstplayer and second player to collaborate for musical performance overnetwork”, which is incorporated in its entirety for all purposes as iffully set forth herein. The providing includes maintaining a referencedata item generated using a musical instrument performed by a user;receiving first user input data generated using a first musicalinstrument, the received first user input data associated with the firstmusical instrument and the musical performance; receiving second userinput data generated using a second musical instrument, the receivedsecond user input data associated with the second musical instrument andthe musical performance; detecting a missing data packet within thereceived second user input data when generating real-time collaborationdata for the musical performance; replacing the missing data packetusing the reference data item to correct the second user input data; andgenerating real-time collaboration data for the musical performancebased on the corrected second user input data.

A system and method for providing exercise in playing a music instrumentis disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled:“System and method for providing exercise in playing a musicinstrument”, which is incorporated in its entirety for all purposes asif fully set forth herein. With prior art solutions there is a commonproblem of inadequate motivation of the user to continue practicing, andinadequate learning of items which are difficult for a specific user.The present solution detects characteristics of the user's play detectedand uses them to provide a suitable program of exercises and to providefeedback which enhances motivation of the user.

An apparatus, system and method are disclosed for teaching musicalinstruction is described in U.S. Pat. No. 9,721,479 to Citron et al.entitled: “Apparatus, system and method for teaching music and other artforms”, which is incorporated in its entirety for all purposes as iffully set forth herein. The invention disclosed includes providing musicinstruction based upon a student's ability and preferred most dynamicefficient method of learning.

A computer implemented method for providing feedback of harmonic contentrelating to a music track is disclosed in U.S. Pat. No. 10,235,898 toRyynanen et al. entitled: “Computer implemented method for providingfeedback of harmonic content relating to music track”, which isincorporated in its entirety for all purposes as if fully set forthherein. The method comprising:

receiving music track information; generating harmonic music trackparameters based on the received music track information; displayingnotation information for a user for performing the music track at agiven time for the music track based on the harmonic music trackparameters; receiving harmonic user content generated by an instrumentperformed by the user, using at least one capturing device; generatingreal-time performance feedback for the user based on comparison of theharmonic user content and the harmonic music track parameters accordingto predefined settings; receiving reference harmonic user content from aplurality of reference users over a public network; adjusting, based onthe reference harmonic user content, at least one of the following: thepredefined settings; and the harmonic music track parameters.

Real-time jamming that is automatically assisted for musicians isdisclosed in U.S. Pat. No. 10,504,498 to Ryynänen et al. entitled:“Real-time jamming assistance for groups of musicians”, which isincorporated in its entirety for all purposes as if fully set forthherein. A real-time audio signal is received of played music that isplayed by at least one person. Beat is tracked of the played music fromthe real-time audio signal and accordingly a time of a next beat ispredicted. At least one of chords; notes; and drum sounds is recognizedfrom the real-time audio signal and repetitions in the played music areaccordingly detected. A next development is predicted in the playedmusic, based on the detected repetitions, including at least one ofchords; notes; and drum sounds that will be played next, and respectivetiming based on the predicted time of the next beat. A real-time outputis produced based on the predicted next development in the played music.

The Learner Interaction Monitoring Systems (LiMS) is a web-basedapplication that can interface with any web-based course deliveryplatform to transform the online learning environment into an activeobserver of learner engagement, and is disclosed in U.S. Pat. No.10,490,096 to Sorenson et al. entitled: “Learner interaction monitoringsystem”, which is incorporated in its entirety for all purposes as iffully set forth herein. The LiMS ‘event capture model’ collects detailedreal-time data on learner behavior in self-directed online learningenvironments, and interprets these data by drawing on behavioralresearch. The LiMS offers education and training managers in corporatecontexts a valuable tool for the evaluation of learner performance andcourse design. By allowing more detailed demonstration of ROI ineducation and training, LiMS allows managers to make the case for webbased courseware that reflects appropriate and evidence-basedinstructional design, rather than budgetary constraints.

A device (1) for monitoring the use accuracy of percussion instruments(5) is disclosed in U.S. Patent Application No. 2015/0310841 toSEMENZATO entitled: “Device for monitoring use accuracy of percussioninstruments”, which is incorporated in its entirety for all purposes asif fully set forth herein. The device includes a sensor (3) intended todetect musical data (T, I) from said instrument (5), an electronicdevice (9) adapted to sample at a sampling frequency Fc the musical data(T, I) detected by said sensor (3), which comprise the instant (T),detected with respect to an initial instant TO taken as a reference, andthe intensity (I) of each stroke that reaches the percussion instrument(5) during an execution of a musical score (PMS) played by a user (100),and adapted to store said musical data (T, I) in a memory device (11).An external computing device (19) having a database (50) containing atleast one musical sample score (PMC) intended to be considered as areference score, or a microprocessor (6) comprised in the electronicdevice (9), compares the musical data (T, I) detected by the sensor (3)during the execution of a played musical score (PMS) by a user (100)with the corresponding musical data (Tc, Ic) of the musical sample score(PMC), and provides the deviation between the measured musical data (T,I) of the played musical score (PMS) and the corresponding ones (Tc, Ic)of the musical sample score (PMC).

Systems and methods capable of providing adaptive and responsiveaccompaniment to music with fixed chord progressions, such as jazz andpop, are disclosed in U.S. Pat. No. 10,032,443 to Braasch et al.entitled: “Interactive, expressive music accompaniment system”, which isincorporated in its entirety for all purposes as if fully set forthherein. A system can include one or more sound-capturing devices, asignal analyzer to analyze captured sound signals, and an electronicsound-producing component that produces electronic sounds as anaccompaniment.

A system (100) for teaching a user to play a musical instrument (114)from musical notation via virtual exercises is described in PatentCooperation Treaty (PCT) International Publication Number WO 2017/037342by LEHTONEN et al. entitled: “System for teaching a user to play amusical instrument from musical notation via virtual exercises and amethod thereof”, which is incorporated in its entirety for all purposesas if fully set forth herein. The system comprising: at least oneelectronic device (102) comprising at least a processing entity (104), amemory entity (106) and a display (108), means for forming or receivingat least one play signal (110) produced by a user on a musicalinstrument (114), the processing entity (104) being arranged to provideat least graphical musical notation content, such as a note or a chord,via the display (108), the processing entity (104) being furtherarranged to obtain at least play signal data via said means for formingor receiving at least one play signal (110), the processing entity (104)being further arranged to execute audio recognition to recognize theplay signal data and compare it with data relating to the presentedgraphical content to at least determine if the play signal correspondsto said graphical content, the processing entity (104) being furtherarranged to assign a score to represent the result of said comparisonand store said score, the processing entity (104) being further arrangedto present the score to the user via the display (108), and todetermine, at least on the basis of said score and other stored scores,further at least graphical musical notation content, to be presented tothe user via the display (108). Corresponding method and computerprogram product are also presented.

Invention that relates to the field of audio recognition, in particularto computer implemented note recognition methods in a gamingapplication, is disclosed in U.S. Pat. No. 9,711,121 to Ahmaniemientitled: “Latency enhanced note recognition method in gaming”, which isincorporated in its entirety for all purposes as if fully set forthherein. Furthermore, the invention relates to improving latency of suchaudio recognition methods. One of the embodiments of the inventiondescribed herein is a method for note recognition of an audio source.The method includes: dividing an audio input into a plurality of frames,each frame having a pre-determined length, conducting a frequencyanalysis of at least a set of the plurality of frames, based on thefrequency analysis, determining if a frame is a transient frame with afrequency change between the beginning and end of the frame, comparingthe frequency analysis of each said transient frame to the frequencyanalysis of an immediately preceding frame and, based on saidcomparison, determining at least one probable pitch present at the endof each transient frame, and for each transient frame, outputting pitchdata indicative of the probable pitch present at the end of thetransient frame.

A workstation system that produces a display presentation of a selectedperformance composition (e.g., a musical composition) responsive tocomposition data and responsive to one or both of input variables and aselected operating mode is disclosed in U.S. Pat. No. 7,157,638 toSitrick entitled: “System and methodology for musical communication anddisplay”, which is incorporated in its entirety for all purposes as iffully set forth herein. The workstation can communicate with one or moreexternal devices, such as other workstations, etc. The display systemprovides for selection of original compositions, creation of derivativecompositions, distribution of compositions, monitoring of eachperformer's performance, group virtual performances, and for local anddistributed retrieval and editing, which for music includes things suchas changing keys, pitch, tempo, and other parameters. The musicalcomposition's transformation can be performed locally or at the centralor distributed music database. The musical composition data can betransposed via a controller, and can be transmitted to a plurality ofthe individual workstations that then display the musical composition.In one embodiment, a display system for use by a plurality of usersprovides a plurality of display presentations of a selected musicalcomposition. The system is comprised of a plurality of individualworkstations, each workstation comprising a communication interfaceproviding for communications with the respective workstation of musicdata representative of the selected musical composition, memory forlocally storing the data responsive to the communications interface, anda display apparatus provides a local visual display presentationrepresentative of the selected musical composition responsive to thestored data. The system further provides for synchronizing thepresentation on the plurality of local visual display presentations ofthe selected musical composition.

An interactive game designed for learning to play a guitar is disclosedin U.S. Pat. No. 8,986,090 to EPSTEIN entitled: “Interactive guitar gamedesigned for learning to play the guitar”, which is incorporated in itsentirety for all purposes as if fully set forth herein. A guitar may beconnected to a computer or other platform, capable of loading music anddisplaying notes and chords and other feedback and visual learning aidson a display screen, allowing a user to read music and play along. Thegoal of the software or interactive game engine is for players to learnhow to play a guitar. Users may operate the game in a number of modeswith different goals, playing mini-games throughout the levels of thegame. The game provides feedback and statistics to help users learn howto play the guitar.

A system and method for learning, composing, accessing and playing musicis disclosed in in Patent Cooperation Treaty (PCT) InternationalPublication Number WO 2015/113360 by Shi entitled: “System and methodfor learning, composing, and playing music with physical objects”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The system includes a plurality of physical objects eachincludes an identifier and assigned a music-related indicator. Thesystem further includes an interactive surface configured to recognizethe identifier and location information relative to the interactivesurface of a physical object placed on top of the interactive surface.Upon a plurality of objects being placed on the interactive surface toform a structural pattern, the processor is configured to derive a musicpiece from the structural pattern.

A system is disclosed in U.S. Pat. No. 7,893,337 to Lenz entitled:“System and method for learning music in a computer game”, which isincorporated in its entirety for all purposes as if fully set forthherein. The system comprising means for receiving a first input from anelectronic device, said first input pertaining to performance of musicby a user, means for receiving a second input, said second inputpertaining at least to music intended to be performed by the user; and acomparison module software executing on a computer and adapted toreceive said first input and to receive said second input, wherein thecomparison module compares the first input from a user to the secondinput to produce at least one indicia of the user's success inperforming the intended music correctly, the comparison module sends toa display module associated with the user information including at leastthe music intended to be performed by the user and the indicia of theuser's success in performing the intended music correctly, and thecomparison module further sends to the display module associated withthe user a timing signal suitable for indicating the speed at which themusic should be shown on the display and played by the user, said timingsignal computed according to one or more tempo modes selectable by theuser, is disclosed.

A music practice feedback method, system, and non-transitory computerreadable medium including a displaying device configured to displaysheet music, a collecting device configured to collect informationrelated to a playing of the sheet music by a plurality of players, and adisplay changing device configured to change a display of the sheetmusic based on said collected information, are disclosed in U.S. Pat.No. 9,672,799 to Kim et al. entitled: “Music practice feedback system,method, and recording medium”, which is incorporated in its entirety forall purposes as if fully set forth herein.

A self-adjusting music scrolling method is disclosed in U.S. Pat. No.7,482,529 to Flamini et al. entitled: “Self-adjusting music scrollingsystem”, which is incorporated in its entirety for all purposes as iffully set forth herein. The self-adjusting music method comprisesproviding a display screen, selecting a music score to be played by amusician, wherein the music score is cataloged and stored in a musicscore database as a first MIDI file, displaying a first portion of theselected music score on the display screen, recording musical notesplayed by a musician with a digital device, storing the recorded musicalnotes in memory as a WAV file, converting the WAV file into a secondMIDI file, comparing the first MIDI file and the second MIDI file with aMIDI comparison algorithm, determining if the first MIDI filesubstantially matches the second MIDI file, automatically adjusting themusic score on the display screen to show a second portion of theselected music score upon determining that the first MIDI filesubstantially matches the second MIDI file, and displaying one or moremistakes detected on the display screen upon determining that the firstMIDI file does not substantially match the second MIDI file.

A system and method in a building or vehicle for an actuator operationin response to a sensor according to a control logic are disclosed inU.S. Patent Application Publication No. 2013/0201316 to Binder et al.entitled: “System and method for server based control”, which isincorporated in its entirety for all purposes as if fully set forthherein. The system comprising a router or a gateway communicating with adevice associated with the sensor and a device associated with theactuator over in-building or in-vehicle networks, and an externalInternet-connected control server associated with the control logicimplementing a PID closed linear control loop and communicating with therouter over external network for controlling the in-building orin-vehicle phenomenon. The sensor may be a microphone or a camera, andthe system may include voice or image processing as part of the controllogic. A redundancy is used by using multiple sensors or actuators, orby using multiple data paths over the building or vehicle internal orexternal communication. The networks may be wired or wireless, and maybe BAN, PAN, LAN, WAN, or home networks.

A method to visually detect and recognize fingering gestures of the lefthand of a guitarist is described in an article by Anne-Marie Burns andMarcelo Wanderley entitled: “Visual Methods for the Retrieval ofGuitarist Fingering”, published Jun. 4-8, 2006 in a Conference Paper of“Proceedings of the 2006 International Conference on New Interfaces forMusical Expression (NIME06)”, Paris, France, which is incorporated inits entirety for all purposes as if fully set forth herein. The methodhas been developed following preliminary manual and automated analysisof video recordings. These first analyses led to some important findingsabout the design methodology of a vision system for guitarist fingering,namely the focus on the effective gesture, the consideration of theaction of each individual finger, and a recognition system not relyingon comparison against a knowledge base of previously learned fingeringpositions. Motivated by these results, studies on three aspects of acomplete fingering system were conducted: the first on finger tracking;the second on strings and frets detection; and the last one on movementsegmentation. Finally, these concepts were integrated into a prototypeand a system for left hand fingering detection was developed.

Sight-reading is the act of performing a piece of music at first sight.This can be a difficult task to master, because it requires extensiveknowledge of music theory, practice, quick thinking, and mostimportantly, a wide variety of musical material. A musician can onlyeffectively sight-read with a new piece of music. This not only requiresmany resources, but also musical pieces that are challenging while alsowithin a player's abilities. A Thesis presented to the Faculty ofCalifornia Polytechnic State University, San Luis Obispo, and publishedJune 2016 by Drew Schulz entitled: “Pianote: A Sight-Reading ProgramThat Algorithmically Generates Music Based On Human Performance”, whichis incorporated in its entirety for all purposes as if fully set forthherein, presents PiaNote, a sight-reading web application for pianiststhat algorithmically generates music based on human performance.PiaNote's goal is to alleviate some of the hassles that pianists facewhen sight-reading. PiaNote presents musicians with algorithmicallygenerated pieces, ensuring that a musician never sees the same piece ofmusic twice. PiaNote also monitors player performances in order tointelligently present music that is challenging, but within the player'sabilities. As a result, PiaNote offers a sight-reading experience thatis tailored to the player. On a broader level, this thesis exploresdifferent methods in effectively creating a sight-reading application.We evaluate PiaNote with a user study involving novice piano players.The players actively practice with PiaNote over three fifteen-minutesessions. At the end of the study, users are asked to determine whetherPiaNote is an effective practice tool that improves both theirconfidence in sight-reading and their sight-reading abilities. Resultssuggest that PiaNote does improve user's sightreading confidence andabilities, but further research must be conducted to clearly validatePiaNote's effectiveness. We conclude that PiaNote has potential tobecome an effective sight-reading application with slight improvementsand further research.

A Brain Automated Chorales (BACh), an adaptive brain-computer systemthat dynamically increases the levels of difficulty in a musicallearning task based on pianists' cognitive workload measured byfunctional near-infrared spectroscopy, is presented in an article inBucknell University Conference Paper published May 2016 by Beste F.Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan,Remco Chang, and Robert JK Jacob entitled: “Learn Piano with BACh: AnAdaptive Learning Interface that Adjusts Task Difficulty based on BrainState”, which is incorporated in its entirety for all purposes as iffully set forth herein. As users' cognitive workload fell below acertain threshold, suggesting that they had mastered the material andcould handle more cognitive information, BACh automatically increasedthe difficulty of the learning task. We found that learners played withsignificantly increased accuracy and speed in the brain-based adaptivetask compared to our control condition. Participant feedback indicatedthat they felt they learned better with BACh and they liked the timingsof the level changes. The underlying premise of BACh can be applied tolearning situations where a task can be broken down into increasinglevels of difficulty.

In consideration of the foregoing, it would be an advancement in the artto provide a method, an apparatus, or a system, for improving theconvenience, the usability, the enjoyment, or the time saving ofenhancing music skills or teaching of playing a musical instrument.Preferably, such methods or systems may be providing an improved,simple, flexible, adaptive, more convenient, automatic, secure,cost-effective, reliable, versatile, time-saving, easy to install, useor monitor, has a minimum part count, portable, handheld, enclosed in asmall or portable housing or vehicular, minimum hardware, and/or usingexisting and available components, protocols, programs and applications,and providing a better user practicing and learning experience, such asinteractive and fun experience with minimum frustration, for enhancingof music skills or teaching of playing a musical instrument

SUMMARY

A method for teaching of playing of a musical instrument to a person,may be used with a client device that may comprise a microphone, asounder (such as a speaker), and a display, may be used with a serverdevice that may communicate over the Internet with the client device,may be used with a first database that may associate a respective skilllevel value to user identifiers, and may be used with a second databasethat may comprise sequences of musical symbols, where each of thesequences may be associated with a respective pace or tempo.

Any method herein may comprise obtaining, by the client device, anidentifier of the person; sending, by the client device to the serverdevice, the person identifier; receiving, by the server device from theclient device, the person identifier; determining, by the server deviceby using the first database, a first skill level value associated withthe received person identifier; selecting, by the server device, a firstsequence of musical symbols from the second database that is associatedwith a first tempo or pace; sending, by the server device to the clientdevice, the selected first sequence of musical symbols and the firstpace; receiving, by the client device from the server device, theselected first sequence of musical symbols and the first pace;displaying, to the person by the display in the client device, thereceived first sequence of musical symbols at a second pace that islower than the first pace and is based on the first pace and on thefirst skill level value; capturing, by the microphone in the clientdevice, a sound from the musical instrument; sending, by the clientdevice to the server device, the captured sound; receiving, by theserver device from the client device, a digital representation of thecaptured sound; analyzing, by the server device, the captured sound andchecking whether the captured sound matches the first sequence ofmusical symbols; determining, by the server device, the amount ofmusical symbols in the first sequence that do not match with thecaptured sound; and updating, by the server device, the skill levelvalue associated with the person identifier in response to the amount ofthe musical symbols that do not match with the captured sound.

Any action or step herein, such as the displaying at the second pacecomprises inducing, by the server device, by the client device, or byany combination thereof, a delay that is according to, or based on, thesecond pace.

Any method herein may be used with a second sequence of symbols that maybe associated with the first pace and may be stored in the serverdevice. The method may further comprise receiving, by the client devicefrom the server device, the second sequence of musical symbols and theupdated skill level value; displaying, to the person by the display inthe client device, the received second sequence of musical symbols at athird pace that is based on the first pace and on the updated skilllevel value; capturing, by the microphone in the client device, a soundfrom the musical instrument; sending, by the client device to the serverdevice, the captured sound; receiving, by the server device from theclient device, a digital representation of the captured sound;analyzing, by the server device, the captured sound, and checkingwhether the captured sound matches the second sequence of musicalsymbols; determining, by the server device, the amount of musicalsymbols in the second sequence that do not match with the capturedsound; and updating, by the server device, the skill level valueassociated with the person identifier in response to the amount of themusical symbols that do not match with the captured sound.

Any device herein, such as the server device, may comprise multipleparts of a musical piece, each represented by a distinct sequence ofsymbols Any method herein may further comprise selecting the secondsequence from the multiple parts sequences. Any device herein, such asthe server device, may comprise at least 2, 3, 4, 5, 7, 10, 12, 15, 20,25, 30, 40, 50, 60, 80, or 100 parts, or may comprise less than 3, 4, 5,7, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, or 200 parts. Any partsherein, and any two parts herein, may be sequential or non-sequential inthe musical piece. Any parts herein, and any two parts herein, may beoverlapping or non-overlapping.

The duration of any one of the parts herein, of each of at least two ofthe parts herein, or of each one of the parts herein, may be at least0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%,30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may be less than 0.2%,0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%,40%, 50%, 60%, 70%, 80%, 90%, or 95% of the musical piece playingduration. Alternatively or in addition, the duration of any one of theparts herein, of each of at least two of the parts herein, or of eachone of the parts herein, may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%,1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%,80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%,3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or95% of the musical piece playing duration. Alternatively or in addition,the duration of any one of the parts herein, of each of at least two ofthe parts herein, or of each one of the parts herein, may be at least atleast 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150,200, 300, 500, or 1000 seconds, or may be less than 2, 3, 5, 8, 10, 12,15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000seconds.

The number of symbols of any one of the parts herein, of each of atleast two of the parts herein, or of each one of the parts herein, maybe at least may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%,3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%,or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%,15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of thenumber of symbols relating in the entire musical piece playing duration.

Any two sequences herein, such as the first and the second sequencesherein, may be part of a musical piece or any part thereof, and anysecond sequence herein may be played immediately after, or motimmediately after, the first sequence in the musical piece.Alternatively or in addition, any two sequences herein, such as thefirst and the second sequences herein, may be overlapping, partiallyoverlapping, or non-overlapping.

Any sequences herein, such as the first and the second sequences herein,may represent the same, or different, playing time. Further, anysequences herein, such as the first and the second sequences herein, mayconsist of, or may comprise, the same, or different, number of symbols.

Any method herein may be used with a threshold, and may further comprisecomparing the amount of the non-matched musical symbols to thethreshold, and acting in response to the number of non-matched musicalsymbols being above the threshold. Any acting herein may comprisestoring, by the server device in the first database, an identifier ofthe first sequence associated with the person identifier. Alternativelyor in addition, any acting herein may comprise sending, by the serverdevice to the client device, a message; and receiving, by the clientdevice from the server device, the message, and any message herein maycomprise identification of the non-matching symbols. Any method hereinmay further comprise notifying, such as by the client device to theperson, in response to the receiving of the message, of the non-matchingsymbols, such as by displaying, to the person on the display, of thenon-matching symbols as marked.

Any threshold herein may be at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25,30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000 non-matchingsymbols, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50,60, 80, 100, 150, 200, 300, 500, 1000 or 2000 non-matching symbols.Alternatively or in addition, any threshold herein may be at least 0.1%,0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%,35%, 40%, 50%, 60%, 70%, 80%, or 90%, of the number of symbols in anysequence, such as in the first sequence, or may be less than 0.2%, 0.3%,0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%,50%, 60%, 70%, 80%, 90%, or 95% of the number of symbols in anysequence, such as in the first sequence.

Any displaying herein may comprise displaying the musical symbols in thefirst sequence one at a time at the second pace, any capturing hereinmay comprise capturing the sound from the musical instrument that may beresponsive to the last displayed music symbol, any analyzing herein ofany captured sound and the checking may comprise analyzing of anyreceived sound and comparing to the last displayed music symbol. Anydetermining herein of the amount of musical symbols may comprisecounting the number of received sounds that may not match with therespective displayed symbols. Any method herein may further comprise, inresponse to counting of a single non-matching symbol, or in response tocounting of two or more non-matching symbols, stopping of the displayingof the musical symbols in the first sequence. Alternatively or inaddition, any method herein may be used with a second sequence ofsymbols that may be associated with the first pace and may be stored inthe server device, and may further comprise in response to counting of asingle non-matching symbol, or in response to counting of two or morenon-matching symbols, displaying, to the person by the display in theclient device, the second sequence of musical symbols at the secondpace.

Alternatively or in addition, any method herein may be used with asecond sequence of symbols that may be associated with the first paceand may be stored in the server device, and may further comprise inresponse to counting of a single non-matching symbol, or in response tocounting of two or more non-matching symbols: sending, by the serverdevice to the client device, a message; receiving, by the client devicefrom the server device, the message; displaying, by the client device onthe display, the received message; obtaining, by the client device fromthe person using an input component, a command; sending, by the clientdevice to the server device, the obtained command; receiving, by theserver device from the client device, the obtained command; and actingin response to the obtained command.

A method may be used with a musical piece, or part thereof, that may becooperatively played by a first musical instrument according to firstsequence of musical symbols, by a second musical instrument according tosecond sequence of musical symbols, and by a third musical instrumentaccording to third sequence of musical symbols. Any method herein may beused with a first client device associated with the first musicalinstrument and operated by a first person that is identified by a firstperson identifier, with a second client device associated with thesecond musical instrument and operated by a second person that isidentified by a second person identifier, and with a third client deviceassociated with the third musical instrument and operated by a thirdperson that is identified by a third person identifier, and each of thefirst, second, and third client devices may comprise a microphone, asounder, and a display and communicates over the Internet with a serverdevice that may store the first, second, and third sequences.

Any method herein may comprise (e.g., synchronously) sending, by theserver device to the first, second, and third client devices,respectively the first, second, and third sequences of musical symbols;receiving, by first, second, and third client devices from the serverdevice, respectively the first, second, and third sequences; displaying,to the first, second, and third persons by the respective display in therespective first, second, and third client devices, the respectivereceived first, second, and third sequences; capturing, by therespective microphone in the first, second, and third client devices,respective first, second, and third sounds from the respective first,second, and third musical instruments; sending, by the first, second,and third client devices to the server device, the respective capturedfirst, second, and third sounds; receiving, by the server device fromthe first, second, and third client devices, the respective capturedfirst, second, and third sounds; analyzing, by the server device, thereceived captured first, second, and third sounds; sending, by theserver device only to the second and third client devices, the receivedcaptured first sound; receiving, by the second and third client devicesfrom the server device, the received captured first sound; emitting, bythe sounder in the second and third client devices, the receivedcaptured first sound; sending, by the server device only to the thirdclient device, the received captured second sound; receiving, by thethird client device from the server device, the received captured secondsound; and emitting, by the sounder in the third client devices, thereceived captured second sound.

Any method herein may further comprise sending, by the server device tothe first client device, the received captured second and third sounds;receiving, by the first client devices from the server device, thereceived captured second and third sounds; and emitting, by the sounderin the first client device, the received captured second and thirdsounds. Any method herein may further comprise sending, by the serverdevice to the second client device, the received captured third sound;receiving, by the second client device from the server device, thereceived captured third sound; and emitting, by the sounder in thesecond client device, the received captured third sound.

Any first musical instrument may consist of, or may comprise, a stringinstrument, and at least one of the first and second musical instrumentsconsists of, or comprises, a string instrument, or at least one of thefirst and second musical instruments may consist of, or may comprise, aninstrument that is not a string instrument. Alternatively or inaddition, any first musical instrument may consist of, or may comprise,a woodwind instrument, and at least one of the first and second musicalinstruments consists of, or comprises, a woodwind instrument, or atleast one of the first and second musical instruments may consist of, ormay comprise, an instrument that is not a woodwind instrument.

Alternatively or in addition, any first musical instrument may consistof, or may comprise, a brass instrument, and at least one of the firstand second musical instruments consists of, or comprises, a brassinstrument, or at least one of the first and second musical instrumentsmay consist of, or may comprise, an instrument that is not a brassinstrument. Alternatively or in addition, any first musical instrumentmay consist of, or may comprise, a percussion instrument, and at leastone of the first and second musical instruments consists of, orcomprises, a percussion instrument, or at least one of the first andsecond musical instruments may consist of, or may comprise, aninstrument that is not a percussion instrument.

Any one of, any of two of, each one of, any first, second, and thirdmusical instruments herein may comprise or may consist of an instrumentthat may be selected from a group that consists of a soprano instrument,an alto instrument, a tenor instruments, a baritone instruments, and abass instruments.

Any musical piece herein may be further cooperatively played by a fourthmusical instrument according to a fourth sequence of musical symbols.Any method herein may further comprise sending, by the server device tothe first, second, or third client device, the fourth sequence, (e.g.,synchronized) with the sending of the first, second, and third sequencesof musical symbols; receiving, by the respective first, second, or thirdclient device from the server device, the fourth sequence; and emitting,by the sounder in the respective first, second, or third client device,a sound according to the fourth sequence that mimics the actual playingof the fourth musical instrument.

Any sending herein may comprise sending to the first, second, and thirdclient devices, and any emitting herein may comprise emitting, by thesounder in each of the first, second, and third client device. Anyfourth instrument herein may be is identical to, may be similar to, ormay be different from, any first, second, or third musical instrument.

Any method herein may further comprise converting the musical symbols inthe fourth sequence into corresponding sound data that may mimic theplaying of the fourth sequence by the fourth musical instrument, and anyemitting herein may comprise sounding of the converted sound data. Anyconverting herein may be performed by any server device herein or by anyclient device herein.

Any method herein may further comprise obtaining, by each of the clientdevices, the respective person identifier; sending, by each of theclient devices to the server device, the respective person identifier;and receiving, by the server device from each of the client device, therespective person identifier. Any server device herein may further storea first skill level value associated with the first person identifier, asecond skill level value associated with the second person identifier,and a third skill level value associated with the third personidentifier, and any musical piece herein may be associated with a firstpace.

Any (e.g., synchronously) sending of each of the first, second, andthird sequences herein, or any displaying of each of the first, second,and third sequences, may be at a second pace that is equal to, or lowerthan, the first pace. Any second pace herein may be based on the first,second, and third skill level values, such as may be based on the lowestvalue of the first, second, and third skill level values.

Each of the skill level values herein may be represented by a numericalvalue, and any second pace herein may be calculated based on a linear ornon-linear function of the numerical values multiplied by the firstpace. Any (e.g., synchronously) sending herein and any displaying hereinat the second pace may comprise inducing a delay that may be accordingto, or may be based on, the second pace. Any inducing of any delayherein may be performed by any server device or by each of the clientdevices.

Any method herein may further comprise checking, by any server device,whether the each of the received captured sounds matches the respectivesequence; determining, by any server device, the amount of musicalsymbols in each of the sequences that do not match with the respectivecaptured sound; and updating, by any server device, the skill levelvalue associated with the respective person identifier in response tothe amount of the musical symbols that do not match in the respectivecaptured sound.

Any musical instrument herein may comprise a keyboard that may consistof a row of keys for being played by the person, and each of the musicalsymbols herein may be associated with one or more of the keys, and anychecking herein may comprise comparing one or more symbols in the firstsequence with the respective sounds when respective one or more keys arepressed by the person. Any analyzing of any captured sound herein maycomprise extracting a feature of the captured sound, and any extractingof any feature herein may comprise, or may be based on, time-domainanalysis, frequency-domain analysis, or time-frequency domain analysis.Any extracted feature herein may comprise, or may consist of, a tonefrequency, an intensity of a tone, or a duration of a tone, a Chromafeature, zero crossings, a peak amplitude, a rise-time, an energy, or atime-delay.

Any analyzing of any captured sound herein may comprise, or may be basedon, a time-domain analysis, or a frequency-domain analysis that maycomprise, or may be based on, a frequency-domain representation that maybe formed using, or may be based on, Fourier series, Fourier transform,Discrete Fourier Transform (DFT), Laplace transform, Z transform, orWavelet transform. Alternatively or in addition, any feature herein maybe extracted by using a Mel-Frequency Analysis, calculatingMel-Frequency Cepstral Coefficients (MFCC), using a Linear PredictiveCoding (LPC), or calculating LPC coefficients.

Any analyzing or checking herein may be based on, or may use, anArtificial Neural Network (ANN) that may be trained to classify anycaptured sound to a musical symbol. Any ANN herein may consist of, maybe based on, or may use, a Feedforward Neural Network (FNN), a RecurrentNeural Network (RNN), or a deep convolutional neural network. Further,Any ANN herein may consist of, may be based on, or may use, at least 3,4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers, or less than 4,5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers.

Any ANN herein may be a Deep Neural Network (DNN), that may comprise, ormay consist of, a Recurrent neural network (RNNs), a Convolutional deepNeural Network (CNNs), an AutoEncoder (AE), a Deep Belief Network (DBN),or a Restricted Boltzmann machine (RBM).

Any database herein, such as the second database, may further comprise asecond sequence of musical symbols for playing of an additional musicalinstrument. Any method herein may further comprises sending, by theserver device to the client device, the second sequence; receiving, bythe client device from the server device, the second sequence; andemitting, by a sounder in the client device, a sound according to thesecond sequence that mimics the playing of the additional musicalinstrument. Any sequence herein, such as the first sequence, may be forplaying of the musical instrument, and any musical piece herein or anypart thereof may comprise any first and second sequences. Any emittingof any sound herein may be at any pace, such as at the second pace, andany displaying herein of the first sequence and the emitting of thesound are synchronized, such as being part of the musical piece.

Any method herein may further comprise converting, by any device such asby the server device or by the client device) any musical symbols in anysequence, such as in the second sequence, into corresponding sound datathat may mimic the playing of the second sequence by the additionalmusical instrument, and any emitting herein may comprise sounding theconverted sound data. Any method herein may further comprise selecting,by any person, any additional musical instrument, and sending, such asby the client device to the server device, the identification of theselected additional musical instrument. Any sounder herein may consistof, may comprise, may use, or may be based on, an electromagneticloudspeaker, a piezoelectric speaker, an Electro-Static Loudspeaker(ESL), a ribbon magnetic loudspeaker, a planar magnetic loudspeaker, abending wave loudspeaker, a piezoelectric transducer, crystal-basedtransducer, a capacitive transducer, or a magnetostrictive transducer.

Any method herein may be used with an additional client device thatcommunicates over the Internet with the server device, and any methodherein may further comprise sending, by the server device to theadditional client device, the received captured sound.

Any method herein may be used with an additional client device that maycommunicate over the Internet with the server device, and may furthercomprise receiving, by the server device from the additional clientdevice, a first sound; sending, by the server device to the clientdevice, the first sound; and emitting, by a sounder in the clientdevice, the first sound. Any sounder herein may consist of, maycomprise, may use, or may be based on, an electromagnetic loudspeaker, apiezoelectric speaker, an Electro-Static Loudspeaker (ESL), a ribbonmagnetic loudspeaker, a planar magnetic loudspeaker, a bending waveloudspeaker, a piezoelectric transducer, crystal-based transducer, acapacitive transducer, or a magnetostrictive transducer.

Any database herein, such as the second database may further comprise asecond sequence of musical symbols for playing of an additional musicalinstrument, and may further comprise sending, by the server device tothe additional client device, the second sequence. Any method herein maybe used with a musical piece, and the first sequence may be for playingof the musical instrument, and the musical piece or a part thereof maycomprise the first and second sequences. Any displaying herein of anysequence, such as the first sequence, and any emitting hereon of anysound, such as the first sound, may be synchronized.

Any method herein may further comprise obtaining, by the client device,a text that may include instructions or cues for playing on the musicalinstrument of the selected first sequence of musical symbols. Any textherein may comprise identification of the musical symbols in theselected first sequence of musical symbols, or may comprise instructionsor cues associated with operating of the musical device. Any musicalinstrument herein may comprise a keyboard that may consist of a row ofkeys, and each of the instructions or cues herein may be associated withone or more keys.

Any method herein may further comprise converting the selected firstsequence of musical symbols to generate the text, and any convertingherein may be performed by the client device. Alternatively or inaddition, any converting herein may be performed by the server device,and any method herein may further comprise sending the text, by theserver device to the client device, and any obtaining by the clientdevice may comprise receiving, by the client device from the serverdevice, the text.

Any method herein may further comprise sounding, such as a human voice,by the client device, the text synchronized with the displaying of thereceived first sequence of musical symbols. Any sounding herein maycomprise, or may be based on, a speech synthesizing or Text-to-Speech(TTS) scheme, that may use, or may be based on, a speech synthesizerthat may be based on, or may use, a concatenative type, using unitselection, diphone synthesis, or domain-specific synthesis.Alternatively or in addition, any sounding herein may comprise, or maybe based on, a speech synthesizing or Text-to-Speech (TTS) scheme, thatmay use, or may be based on, a speech synthesizer that may be based on,or may use, a formant type or may be Hidden Markov models (HMM) based.Alternatively or in addition, any sounding herein may comprise, or maybe based on, a speech synthesizing or Text-to-Speech (TTS) scheme, thatmay use, or may be based on, stored prerecorded human voice.

Any method herein may further comprise haptic notifying, by the clientdevice, in response to the selected first sequence of musical symbols.Any haptic notifying herein may comprise, may use, or may be based on,cutaneous, kinaesthetic, or orhaptic technology. Any haptic notifyingherein may use vibration that may be produced by an Eccentric RotatingMass (ERM) actuator, a Linear Resonant Actuator (LRA), piezoelectricactuators, an unbalanced motor, a loudspeaker, an ultrasound transducer,or an air vortex ring.

Any device herein, such as the client device, may comprise a projector,and any displaying herein may comprise illuminating, by the projector,elements of the musical instrument that may be associated with thereceived first sequence of musical symbols.

Any musical instrument herein may further comprise a keyboard that mayconsist of a row of keys, and any displaying herein may compriseilluminating, by the projector, on one or more keys that may beassociated with the received first sequence of musical symbols. Anyprojector herein may comprise, or may consist of, an Eidophor projector,Liquid Crystal on Silicon (LCoS or LCOS) projector, LCD projector, MEMSprojector, or Digital Light Processing (DLP™) projector.

Any method herein may further comprise notifying a feedback message,such as a text message, to the person in response to the amount of themusical symbols that do not match with the captured sound, or inresponse to updating the skill level value.

Any notification herein may comprise sounding as a human voice, by theclient device, the text message, and any sounding herein may comprise,or may be based on, a speech synthesizing or Text-to-Speech (TTS)scheme. Any speech synthesizing or Text-to-Speech (TTS) herein may use,or may be based on, a speech synthesizer that may be based on, or mayuse, a concatenative type, using unit selection, diphone synthesis,domain-specific synthesis, a formant type or is Hidden Markov models(HMM) based. Alternatively or in addition, any notifying herein maycomprise displaying, to the person by the display, the message.

Any method herein may be used with an electronic musical instrument thatmay comprise a port for outputting messages in response to actions ofthe person, and each of the musical symbols herein, such as in theselected first sequence, may correspond to a respective person action.Any method herein may comprise receiving, from the electronic musicalinstrument, the person actions; sending, by the client device to theserver device, the received person actions; receiving, by the serverdevice from the client device, the person actions; analyzing, by theserver device, the person actions and checking whether the personactions match the first sequence of musical symbols; determining, by theserver device, the amount of musical symbols in the first sequence thatdo not match with the person actions; and updating, by the serverdevice, the skill level value associated with the person identifier inresponse to the amount of the musical symbols that do not match with theperson actions.

Any receiving herein from any electronic musical instrument of theperson actions may comprise receiving serial data stream according to anindustry standard that may be according to, or may be based on, MIDIstandard protocol. Further, any checking whether the person actionsmatch the first sequence of musical symbols may be according to, or maybe based on, MIDI standard protocol. Any electronic musical instrumentherein may comprise, or may consist of, a MIDI controller or a MIDIkeyboard.

Any port herein may comprise, or may consist of, a first antenna, andany receiving herein from the electronic musical instrument of theperson actions may comprise receiving over a wireless communication by asecond antenna using a wireless transceiver in the client device, andany wireless communication herein may comprise a Wireless Personal AreaNetwork (WPAN), any wireless transceiver herein may comprise a WPANtransceiver, and any of the antennas herein may comprise an WPANantenna. Any WPAN herein may be according to, may be compatible with, ormay be based on, Bluetooth™, Bluetooth Low Energy (BLE), or IEEE802.15.1-2005 standards, or wherein the WPAN is a wireless controlnetwork that may be according to, or may be based on, Zigbee™, IEEE802.15.4-2003, or Z-Wave™ standards.

Alternatively or in addition, any port herein may comprise, or mayconsist of, a first connector, and any receiving herein from theelectronic musical instrument of the person actions may comprisereceiving over a wired point-to-point cable by a second connector in theclient device. Any of the connectors herein may be a MIDI DIN connector,and any communication over the cable herein may be according to, or maybe based on, MIDI standard protocol. Alternatively or in addition, anyof the connectors herein may be a Universal Serial Bus (USB) connector,and any communication over any cable herein may be according to, or maybe based on, USB 2.0 or USB 3.0 standard.

Any selecting herein any sequence of musical symbols, such as the firstsequence of musical symbols, may comprises selecting, by the serverdevice, a second sequence of musical symbols from the second database;and creating, by the server device, the first sequence of musicalsymbols from the selected second sequence of musical symbols by changingan arrangement of the selected second sequence of musical symbols forproducing a simplified arrangement as the first sequence of musicalsymbols. Any changing herein of any arrangement to a simplifiedarrangement may be based on, or may be according to, a skill levelvalue, such as the first skill level value

Any sequence of musical symbols herein, such as the first sequence ofmusical symbols, may be associated with a distinct complexity ordifficulty related value that may be equal to, or may be based on, anyskill level value, such as the first skill level value, associated withthe person identifier. Any method herein may be used with multiplemusical pieces, and any selecting herein, such as by the server deviceor by the client device, of any sequence of musical symbols from anydatabase, such as selecting of the second sequence of musical symbolsfrom the second database, may comprises selecting of a musical piecefrom the multiple musical pieces. Any sequence of musical symbolsherein, such as the second sequence of musical symbols, may be anon-simplified original sequence of musical symbols of the selectedmusical piece.

Any method herein may further comprise, responsive to any updating ofthe skill level value, creating, by any device such as the server deviceor the client device, a third sequence of musical symbols from theselected second sequence of musical symbols, such as by changing anarrangement of the selected second sequence of musical symbols forproducing a simplified arrangement as the third sequence of musicalsymbols.

Any changing herein of any arrangement to a simplified arrangement maybe based on, or may be according to, any skill level value, such as theupdated skill level value. Any sequence of musical symbols herein, suchas the third sequence of musical symbols, may be associated with adistinct complexity or difficulty related value that may be equal to, ormay be based on, any skill level value that may be associated with theperson identifier, such as the first skill level value that isassociated with the person identifier.

Any method herein may further comprise, sending, by the server device tothe client device, the third sequence of musical symbols; receiving, bythe client device from the server device, the third sequence of musicalsymbols; displaying, to the person by the display in the client device,the third sequence of musical symbols; capturing, by the microphone inthe client device, a sound from the musical instrument; sending, by theclient device to the server device, a digital representation of capturedsound; receiving, by the server device from the client device, thedigital representation of the captured sound; analyzing, by the serverdevice, the captured sound and checking whether the captured soundmatches the third sequence of musical symbols; determining, by theserver device, the amount of musical symbols in the third sequence thatdo not match with the captured sound; and updating, by the serverdevice, the skill level value associated with the person identifier inresponse to the amount of the musical symbols that do not match with thecaptured sound.

Any producing herein of any simplified arrangement, such as the first orthird sequence of musical symbols, may comprise changing a hand orfinger motion relative to a geometry of the playing mechanism of themusical instrument; changing a playing speed, a note density, a DistinctStroke Rate (DSR), a Pitch Entropy (PE), a Hand Displacement Rate (HDR),a Hand Stretch (HS), a PolyPhony Rate (PPR), an Altered Note Rate (ANR),or a fingering complexity (FCX); or a changing fingers positioning, atransition between symbols, a tempo, or decorations. Alternatively or inaddition, any producing herein of any simplified arrangement, such asthe first or third sequence of musical symbols, may comprise changingthe physical difficulty of playing the sequence based on the physicalstructure of the musical instrument; changing a rhythmic difficulty thatcomprises a rate of playing and the irregularity of the musical symbolsin time; changing a harmonic difficulty that comprises a combinationcomplexity of simultaneously or sequentially played musical symbols;changing an expressivity difficulty that comprises a need to control andvary the loudness and timbre of the musical symbols; or changing thenumber of simultaneously played symbols or chords.

Any selecting herein, such as any selecting of a musical piece or a partthereof, may be based on, or may use, a First-In-First-Out (FIFO)scheme, a Last-In-First-Out (LIFO) scheme, a sequential or cyclicselection, a random selection, or any combination thereof. Alternativelyor in addition, any selecting herein may be based on, or may use, arandom selection that may be based on, or may use, one or more randomnumbers generated by a random number generator. Any random numbergenerator herein may be hardware based, and may be based on, or may use,thermal noise, shot noise, nuclear decaying radiation, photoelectriceffect, or quantum phenomena. Alternatively or in addition, any randomnumber generator may be software based, such as based on executing analgorithm for generating pseudo-random numbers.

Any method herein may be used with a plurality of musical pieces, andany database herein, such as the second database, may comprise multiplesequences of musical symbols for each of the plurality of musicalpieces, and a distinct complexity or difficulty related value may beassociated with each of the multiple sequences of musical symbols ofeach of the plurality of musical pieces. Any selecting herein maycomprise selecting of a musical piece from the plurality of musicalpieces, and may further comprise selecting of a first sequence from themultiple sequences of musical symbols of the selected musical piece.Each of the plurality of musical pieces may be associated with anassociated sequence of musical symbols that may be an originalnon-simplified version of the respective musical piece.

Any database herein, such as the second database, may further comprise adistinct complexity or difficulty related value associated with each ofthe multiple sequences of musical symbols, and any selecting herein maybe based on comparing the complexity or difficulty values with the firstskill level value associated with the person identifier. Each of theskill level values herein may be a numerical value, and each of thecomplexity or difficulty values may be a numerical value, and anyselecting herein may be based on, or may use, selecting of a sequence ofmusical symbols that may be associated with a complexity or difficultyvalue that may be equal to, or lower than, the first skill level valueassociated with the person identifier.

Any method herein may further comprise estimating or calculating thecomplexity or difficulty related value that may be associated with atleast one of, or all of, the multiple sequences of musical symbols. Anyestimating or calculating of the complexity or difficulty related valueherein may be performed by the server device, by the client device, orany combination thereof. Further, any estimating or calculating of thecomplexity or difficulty related value herein may be based on estimatedcomplexity or difficulty of playing the at least one of the multiplesequences of musical symbols on the musical instrument.

Any method herein may further comprise, responsive to any updating theskill level value, selecting, such as based on comparing the complexityor difficulty values with the updated skill level value, a secondsequence of musical symbols from the multiple sequences of musicalsymbols. Any two sequences herein, such as the first and secondsequences, may be representations of the same musical piece, and thesecond sequence may follow the first sequence when playing the musicalpiece. Any method herein may further comprise sending, by the serverdevice to the client device, the selected second sequence of musicalsymbols; receiving, by the client device from the server device, theselected second sequence of musical symbols; and displaying, to theperson by the display in the client device, the received second sequenceof musical symbols.

Any estimating or calculating herein of the complexity or difficultyrelated value of the at least one of the multiple sequences of musicalsymbols may be based on a required hand or finger motion and thegeometry of the playing mechanism of the musical instrument, a playingspeed, a note density, a pitch entropy, a hand displacement rate, a handstretch, or fingering complexity.

Alternatively or in addition, any estimating or calculating herein ofthe complexity or difficulty related value of the at least one of themultiple sequences of musical symbols may be based on, or may comprise,associating identifying phrases in the sequence with a weightedimportance value; estimating or calculating a distance of notes, tempo,or ease of interpretation; estimating or calculating fingerspositioning, transition between symbols, tempo, or decorations;estimating or calculating motoric difficulty that comprises the physicaldifficulty of playing the sequence based on the physical structure ofthe musical instrument; rhythmic difficulty that comprises a rate ofplaying and the irregularity of the musical symbols in time; harmonicdifficulty that comprises a combination complexity of simultaneously orsequentially played musical symbols; or expressivity difficulty thatcomprises a need to control and vary the loudness and timbre of themusical symbols.

Alternatively or in addition, any estimating or calculating herein ofthe complexity or difficulty related value of the at least one of themultiple sequences of musical symbols may be based on, or may comprise,extracting a feature of the sequence of musical symbols, and anyextracting of any feature herein may comprise, or may be based on, timedomain analysis, frequency domain analysis, or time-frequency domainanalysis, using a Mel-Frequency Analysis, calculating Mel-FrequencyCepstral Coefficients (MFCC), using a Linear Predictive Coding (LPC), orcalculating LPC coefficients. Any extracted feature herein may comprise,or may consist of, a tone frequency, an intensity of a tone, a durationof a tone, a Chroma feature, zero crossings, a peak amplitude, arise-time, an energy, a time-delay, a frequency-domain representationthat is formed using, or based on, Fourier series, Fourier transform,Discrete Fourier Transform (DFT), Laplace transform, Z transform,Wavelet transform,

Any sequence of musical symbols herein may be based on, may consist of,or may comprise, instructions for playing a musical piece or a partthereof, and any pace herein may be based on, may consist of, or maycomprise, a tempo that may be associated with the musical piece. Anytempo or pace herein may be based on, may consist of, or may comprise,Larghissimo, Adagissimo, Grave, Largo, Lento, Larghetto, Adagio,Adagietto, Andante, Andantino, Marcia moderato, Andante moderato,Moderato, Allegretto, Allegro moderato, Allegro, Molto Allegro, Vivace,Vivacissimo, Allegrissimo, Allegro vivace, Presto, or Prestissimo.Alternatively or in addition, any tempo herein may be above 24 beats perminute (bpm), may be above 200 bpm, or may be within one of the rangesof 25-45 bpm, 40-60 bpm, 45-60 bpm, 60-66 bpm, 66-76 bpm, 72-76 bpm,70-80 bpm, 76-108 bpm, 80-108 bpm, 83-85 bpm, 92-98 bpm, 98-112 bpm,102-110 bpm, 116-120 bpm, 120-156 bpm, 124-156 bpm, 156-176 bpm, 172-176bpm, or 168-200 bpm.

Any pace or tempo herein that is based on another tempo or pace, may beless than 95%, 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%,35%, 30%, 25%, 20%, 15%, or 10% of the another pace or tempo.Alternatively or in addition, any pace or tempo herein that is based onanother tempo or pace, may be more than 90%, 85%, 80%, 75%, 70%, 65%,60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, or 5% of theanother pace or tempo. Alternatively or in addition, any pace or tempovalue herein is calculated based on a linear or non-linear function of anumerical value multiplied by another pace or tempo value, and thenumerical value may be a function of, or a representation of, a skilllevel such as the first skill level.

Any musical instrument herein may consists of, or may comprise, aSoprano instrument such as a flute, a violin, a soprano saxophone, atrumpet, a clarinet, an oboe, or a piccolo. Alternatively or inaddition, any musical instrument herein may consists of, or maycomprise, an Alto instrument, such as an alto saxophone, an French horn,an English horn, a viola, or an alto horn. Alternatively or in addition,any musical instrument herein may consists of, or may comprise, a Tenorinstrument, such as a trombone, a tenoroon, a tenor saxophone, a tenorviolin, a guitar, or a tenor drum. Alternatively or in addition, anymusical instrument herein may consists of, or may comprise, a Baritoneinstrument, such as a bassoon, a baritone a saxophone, a bass clarinet,a cello, a baritone horn, or an euphonium. Alternatively or in addition,any musical instrument herein may consists of, or may comprise, a Bassinstrument, such as a double bass, a bass guitar, a contrabassoon, abass saxophone, a tuba, or a bass drum.

Any musical instrument herein may consist of, or may comprise, a stringinstrument that produces sound by means of vibrating strings, such as aguitar, an electric bass, a violin, a viola, a cello, a double bass, abanjo, a mandolin, an ukulele, or a harp. Any string instrument hereinmay consist of, or may comprise, a lute instrument in which the stringsare supported by a neck and a bout, a harp instrument, in which thestrings are contained within a frame, or a zither instrument, in whichthe strings are mounted on a body. Further, any string instrument hereinbe configured to be played by plucking, bowing, or striking.Furthermore, the string instrument any string instrument herein may bean acoustic instrument, or alternatively may comprise an electricamplification.

Any musical instrument herein may consist of, or may comprise, awoodwind instrument that produces sound when the player blows airagainst a sharp edge or through a reed, causing the air within itsresonator to vibrate, such as a flute, a piccolo, an oboe, an Englishhorn, a clarinet, a bass clarinet, a bassoon, a contrabassoon, or asaxophone. Any woodwind instrument herein may consist of, or maycomprise, a flute that is configured to produce sound when air is blownacross an edge, and any flute herein may consist of, or may comprise, anopen flute, in which the player's lips form a stream of air which goesdirectly from the players lips to the edge, or a closed flute, in whichin which the musical instrument has a channel to form and direct the airstream over an edge. Alternatively or in addition, any woodwindinstrument herein may consist of, or may comprise, a reed instrumentthat is configured to produce sound by focusing air into a mouthpiecewhich then causes a reed, or reeds, to vibrate, and any reed instrumentherein may consist of, or may comprise, a single reed instrument thatuses a reed, which is a thin-cut piece of cane or plastic that is heldagainst the aperture of a mouthpiece with a ligature, so that when airis forced between the reed and the mouthpiece, the reed vibrates tocreate a sound, a double reed instrument that uses two precisely cutsmall pieces of cane that are joined together at the base and areinserted into the top of the instrument to vibrate as air is forcedbetween the two pieces, or a capped double reed instrument configured tovibrate when the player blows through a hole in a cap that covers thereed.

Any musical instrument herein may consist of, or may comprise, a brassinstrument that produces sound by sympathetic vibration of air in atubular resonator in sympathy with the vibration of the player's lips,such as a Trumpet, a Cornet, a Horn, a Trombone, a Saxhorn, or a Tuba.Any brass instrument herein may consist of, or may comprise, a valvedbrass instrument that uses a set of valves operated by the player'sfingers that introduce additional tubing, or crooks, into the instrumentfor changing its overall length, or a slide brass instrument that uses aslide to change the length of tubing.

Any musical instrument herein may consist of, or may comprise, apercussion instrument that is configured to produce sound by beingstruck or scraped by a beater, such as a timpani, a snare drum, a bassdrum, cymbals, a triangle, a tambourine, a glockenspiel, or a xylophone.Any percussion instrument herein may consist of, or may comprise, apitched percussion instrument that produces notes with an identifiablepitch, or an unpitched percussion instrument that produces notes orsounds in an indefinite pitch.

Any musical instrument herein may comprise, or may be based on, anacoustic, stringed musical instrument, configured so that the stringsare struck by wooden hammers that are coated with a softer material, andthe musical instrument may further comprise a keyboard that may consistof a row of keys, may be configured so that the performer presses downor strikes with the fingers and thumbs of both hands to cause thehammers to strike the strings. Any musical instrument herein maycomprise, or may consist of, a piano, that may comprise, or may consistof, a grand piano, an upright piano, or an electronic piano.

Any playing of any musical instrument herein may consists of, maycomprises, or may be supplemented with, a vocal music, that may be withor without instrumental accompaniment, and may use lyrics, such assinging.

Any sequence herein may include musical symbols, and each of the musicalsymbols in the sequence may be according to a musical notationconvention. Further, any sequence herein may be part of, or may consistof, a music sheet. Any sequence herein may include one or more clefsthat may define the pitch range or the tessitura of the symbol on whichit is placed, one or more musical notes that may denote a musical soundor the pitch and duration of a musical sound, one or more accidentalsthat may denote a pitch or a pitch class, one or more key signaturesthat may define the prevailing key of the music that follows, one ormore time signatures that may define the meter of the music, one or morelines or a note relationships, one or more dynamics that may indicatethe relative intensity or volume of a musical line, one or morearticulations that may specify how to perform individual notes within aphrase or passage, one or more ornaments that may modify the pitchpattern of an individual note, one or more octave signs, repetitions orcodas, one or more pedal marks, or any combination thereof.

Any method herein may be used with a threshold, and any updating hereinmay comprise raising or lowering of the skill level value associatedwith the person identifier in response to the amount of the non-matchingmusical symbols being less or more than the threshold. Any thresholdherein may be at least 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50 non-matchingsymbols, or may be less than 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50non-matching symbols. Alternatively or in addition, any threshold hereinmay be at least 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%, 5%, 8%, 10%, 15%,20%, 30%, or 50% non-matching symbols out of the total number of symbolsin the sequence, or may be less than 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%,5%, 8%, 10%, 15%, 20%, 30%, or 50% non-matching symbols out of the totalnumber of symbols in the sequence. Alternatively or in addition, anymethod herein may be used with low and high thresholds, and any updatingherein may comprise unchanging the skill level value associated with theperson identifier in response to the amount of the non-matching musicalsymbols being more than the low threshold and less than the highthreshold.

Any sequence herein of any musical symbols may represents an entire of,or part of, a musical piece. Any musical piece herein may includesymbols for multiple musical instruments, and any sequence herein mayrepresent or include symbols adapted only for one specific musicalinstrument. Any musical piece herein may comprise, or may consist of asong, a vocal music work, or a classical music work, such as an Aria, anCadenza, an Concerto, an Movement, an Overture, an Opera, an Sonata, anChamber music, or an Symphony. Alternatively or in addition, any musicalpiece herein may comprise, or may consist of, a popular music work, suchas a song, a dance, or a Funk, Country, Latin, Reggae, Hip-hop, or Polkamusic genre.

Any sequence of the musical symbols herein may represent a part of themusical piece that may be associated with a duration of playing theentire musical piece, or that may define a number of symbols. Anyduration of any part herein is at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%,1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%,80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%,3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or95%, of the duration of the playing time of the musical piece.Alternatively or in addition, any duration of playing any sequenceherein is at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60,80, 100, 150, 200, 300, 500, or may be less than 2, 3, 5, 8, 10, 12, 15,20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000seconds. Alternatively or in addition, any sequence herein may comprise,or may consist of, at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%,3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%,or may comprise, or may consist of, less than 0.2%, 0.3%, 0.5%, 0.8%,1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%,80%, 90%, or 95%, of the number of symbols of the entire musical piece.Alternatively or in addition, any sequence herein may comprise, or mayconsist of, at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60,80, 100, 150, 200, 300, 500, or 1000 symbols, or less than 2, 3, 5, 8,10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or2000 symbols.

Any microphone herein may be configured to respond to audible orinaudible sound, and may be an omnidirectional, unidirectional, orbidirectional microphone. Any microphone herein may be based on thesensing the incident sound-based motion of a diaphragm or a ribbon.Alternatively or in addition, any microphone herein may consist of, ormay comprise, a condenser, an electret, a dynamic, a ribbon, a carbon,an optical microphone, or a piezoelectric microphone. Further, anymicrophone herein may consist of, may comprise, or may be based on, amicrophone array for improving directivity.

Any display or any display screen herein may consist of, or maycomprise, a monochrome, grayscale or color display and consists of anarray of light emitters or light reflectors, or a projector that isbased on an Eidophor, Liquid Crystal on Silicon (LCoS or LCOS), LCD,MEMS or Digital Light Processing (DLP™) technology. Any projector hereinmay consist of, or may comprise, a virtual retinal display. Further, anydisplay or any display screen herein may consist of, or may comprise, a2D or 3D video display that may support Standard-Definition (SD) orHigh-Definition (HD) standards, and may be capable of scrolling, static,bold or flashing the presented information.

Alternatively or in addition, any display or any display screen hereinmay consist of, or may comprise, an analog display having an analoginput interface supporting NTSC, PAL or SECAM formats, and the analoginput interface may include RGB, VGA (Video Graphics Array), SVGA (SuperVideo Graphics Array), SCART or S-video interface. Alternatively or inaddition, any display or any display screen herein may consist of, ormay comprise, a digital display having a digital input interface thatmay include IEEE1394, FireWire™, USB, SDI (Serial Digital Interface),HDMI (High-Definition Multimedia Interface), DVI (Digital VisualInterface), UDI (Unified Display Interface), DisplayPort, DigitalComponent Video or DVB (Digital Video Broadcast) interface.Alternatively or in addition, any display or any display screen hereinmay consist of, or may comprise, a Cathode-Ray Tube (CRT), a FieldEmission Display (FED), an Electroluminescent Display (ELD), a VacuumFluorescent Display (VFD), or an Organic Light-Emitting Diode (OLED)display, a passive-matrix (PMOLED) display, an active-matrix OLEDs(AMOLED) display, a Liquid Crystal Display (LCD) display, a Thin FilmTransistor (TFT) display, an LED-backlit LCD display, or an ElectronicPaper Display (EPD) display that may be based on Gyricon technology,Electro-Wetting Display (EWD), or Electrofluidic display technology.Alternatively or in addition, any display or any display screen hereinmay consist of, or may comprise, a laser video display that is based ona Vertical-External-Cavity Surface-Emitting-Laser (VECSEL) or aVertical-Cavity Surface-Emitting Laser (VCSEL). Further, any display orany display screen herein may consist of, or may comprise, a segmentdisplay based on a seven-segment display, a fourteen-segment display, asixteen-segment display, or a dot matrix display, and may be operativeto display digits, alphanumeric characters, words, characters, arrows,symbols, ASCII, non-ASCII characters, or any combination thereof.

Any sound herein may be audible or inaudible (or both), and may beomnidirectional, unidirectional, bidirectional, or provide otherdirectionality or polar patterns. Any sounder herein may be anelectromagnetic loudspeaker, a piezoelectric speaker, an Electro-StaticLoudspeaker (ESL), a ribbon or planar magnetic loudspeaker, or a bendingwave loudspeaker. Any sounder herein may consist of, may comprise, mayuse, or may be based on, an electric sound source that may convertelectrical energy into sound waves, and the electric sound source may beconfigured to emit an audible or inaudible sound using omnidirectional,unidirectional, or bidirectional pattern. Further, any sounder hereinmay convert electrical energy to sound waves transmitted through theair, an elastic solid material, or a liquid, usually by means of avibrating or moving ribbon or diaphragm.

Further, the electric sound source may consist of, may comprise, mayuse, or may be based on, an electromagnetic loudspeaker, a piezoelectricspeaker, an electrostatic loudspeaker (ESL), a ribbon magneticloudspeaker, a planar magnetic loudspeaker, or a bending waveloudspeaker. Alternatively or in addition, the electric sound source mayconsist of, may comprise, may use, or may be based on, anelectromechanical scheme or a ceramic-based piezoelectric effect.Alternatively or in addition, the electric sound source may consist of,may comprise, may use, or may be based on, an ultrasonic transducer thatmay be a piezoelectric transducer, crystal-based transducer, acapacitive transducer, or a magnetostrictive transducer.

Any server herein may be storing, operating, or using, a serveroperating system, which may consist of, may comprise, or may be basedon, one out of Microsoft Windows Server®, Linux, or UNIX, or may consistof, may comprise, or may be based on, one out of Microsoft WindowsServer® 2003 R2, 2008, 2008 R2, 2012, or 2012 R2 variant, Linux™ orGNU/Linux-based Debian GNU/Linux, Debian GNU/kFreeBSD, Debian GNU/Hurd,Fedoram, Gentoo™, Linspire™, Mandriva, Red Hat® Linux, SuSE, andUbuntu®, UNIX® variant Solaris™, AIX®, Mac™ OS X, FreeBSD®, OpenBSD, andNetBSD®.

Any software or firmware herein may comprise an operating system thatmay be a mobile operating system. The mobile operating system mayconsist of, may comprise, may be according to, or may be based on,Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Androidversion 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean),Android version 4.4 (KitKat), Android version 6.0 (Marshmallow), Androidversion 7.0 (Nougat), Android version 8.0 (Oreo), Android version 9.0(Pie), Android 10, Android 11, Apple iOS version 3, Apple iOS version 4,Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, Apple iOSversion 8, Apple iOS version 9, Apple iOS version 10, Apple iOS version11, Apple iOS version 12, Apple iOS version 13, Apple iOS version 14,Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8,Microsoft Windows® Phone version 9, or Blackberry® operating system. AnyOperating System (OS) herein, such as any server or client operatingsystem, may consists of, include, or be based on a real-time operatingsystem (RTOS), such as FreeRTOS, SafeRTOS, QNX, VxWorks, orMicro-Controller Operating Systems (μC/OS).

Any device herein may consist of, may comprise, may be part of, or mayintegrated with, a client device that may comprise a single enclosurethat may house the microphone and the display. Any device herein mayconsist of, may comprise, may be part of, or may integrated with, aclient device, that may consist of, may comprise, may be part of, or mayintegrated with, a hand-held enclosure, a portable enclosure, or asurface mountable enclosure. Further, any client device herein mayconsist of, may comprise, may be part of, or may be integrated with, anotebook computer, a laptop computer, a media player, a cellulartelephone, a tablet device, or a smartphone, which may consist of, maycomprise, or may be based on, an Apple iPhone 12 or a Samsung GalaxyS20.

Any device herein, such as any client device herein, may be configuredor shaped to be wearable on a person, such as on an organ of the personhead, and the organ may be an eye, ear, face, cheek, nose, mouth, lip,forehead, or chin. Any device herein may include an enclosure that maybe constructed to have a form substantially similar to, may beconstructed to have a shape allowing mounting or wearing identical orsimilar to, or may be constructed to have a form to at least in partsubstitute for, headwear, eyewear, or earpiece.

Any headwear herein may consist of, may be structured as, or maycomprise, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, ahelmet, a hood, a mask, a turban, a veil, or a wig. Any eyewear hereinmay consist of, may be structured as, or may comprise, glasses,sunglasses, a contact lens, a blindfold, or a goggle. Any earpieceherein may consist of, may be structured as, or may comprise, a hearingaid, a headphone, a headset, or an earplug. Any enclosure herein may bepermanently or releasably attachable to, or is part of, a clothing pieceof a person, and any attaching herein may use taping, gluing, pinning,enclosing, encapsulating, a pin, or a latch and hook clip. Any clothingpiece herein may be a top, bottom, or full-body underwear, or aheadwear, a footwear, an accessory, an outwear, a suit, a dress, askirt, or a top. Any device herein may comprise an annular memberdefining an aperture therethrough that is sized for receipt therein of apart of a human body. Any human body part herein may be part of a humanhand that may consist of, or may comprise, an upper arm, elbow, forearm,wrist, or a finger. Alternatively or in addition, any human body partherein may be part of a human head or neck that may consist of, or maycomprise, a forehead, ear, skull, or face. Alternatively or in addition,any human body part herein may be a part of a human thorax or abdomenthat may consist of, or may comprise, a waist or hip. Alternatively orin addition, any human body part herein may be part of a human leg orfoot that may consist of, or may comprise, a thigh, calf, ankle, instep,knee, or toe.

Any method herein may be in combination with a Virtual Reality (VR)system that simulates a virtual environment to the person, and anycommunication herein with the VR system may be wired or wireless. Any VRsystem herein may comprises a Head-Mounted Display (HMD), and any deviceherein, such as any client device herein, may comprise, may be part of,may consist of, or may be integrated with, the HMD. Any method hereinmay further be used in an Augmented Reality (AR) system or a SpatialAugmented Reality (SAR), that simulates a virtual environment to theperson, and any display herein may comprise, may be implemented as, ormay be integrated with, an Head-Mounted Display (HMD), a Head-Up Display(HUD), contact lenses, a Virtual Retinal Display (VRD).

Any integration herein, such as integration of any client device hereinwith any musical instrument herein, may involve sharing a component,housing in same enclosure, sharing same processor, mounting onto samesurface, or sharing a same connector, which may be a power connector forconnecting to a power source. Alternatively or in addition, theintegration may involve sharing the same connector for being poweredfrom same power source, or the integration may involve sharing samepower supply.

Any communication herein, such as between a client device and a serverdevice, may be over the Internet. Alternatively or in addition, thecommunication may be over the Internet via a wireless network, and theclient device may comprise an antenna and a wireless transceiver coupledto the antenna for communication via the wireless network.

Any wireless network herein may comprise a Wireless Wide Area Network(WWAN), any wireless transceiver herein may comprise a WWAN transceiver,and any antenna herein may comprise a WWAN antenna. Any WWAN herein maybe a wireless broadband network. The WWAN may be a WiMAX network, theantenna may be a WiMAX antenna and the wireless transceiver may be aWiMAX modem, and the WiMAX network may be according to, compatible with,or based on, IEEE 802.16-2009. Alternatively or in addition, the WWANmay be a cellular telephone network, the antenna may be a cellularantenna, and the wireless transceiver may be a cellular modem, where thecellular telephone network may be a Third Generation (3G) network thatmay use a protocol selected from the group consisting of UMTS W-CDMA,UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, and GSMEDGE-Evolution, or the cellular telephone network may use a protocolselected from the group consisting of a Fourth Generation (4G) networkthat use HSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be basedon IEEE 802.20-2008.

Any wireless network herein may comprise a Wireless Personal AreaNetwork (WPAN), the wireless transceiver may comprise a WPANtransceiver, and the antenna may comprise a WPAN antenna. The WPAN maybe according to, compatible with, or based on, Bluetooth™, Bluetooth LowEnergy (BLE), or IEEE 802.15.1-2005 standards, or the WPAN may be awireless control network that may be according to, or may be based on,Zigbee™, IEEE 802.15.4-2003, or Z-Wave™ standards. Any wireless networkherein may comprise a Wireless Local Area Network (WLAN), \ the wirelesstransceiver may comprise a WLAN transceiver, and the antenna maycomprise an WLAN antenna. The WLAN may be according to, may becompatible with, or may be based on, a standard selected from the groupconsisting of IEEE 802.11-2012, IEEE 802.11a, IEEE 802.11b, IEEE802.11g, IEEE 802.11n, and IEEE 802.11ac. Any wireless network hereinmay be over a licensed or unlicensed radio frequency band that may be anIndustrial, Scientific and Medical (ISM) radio band.

Any method herein may be performed partly or in full by any device, suchas any client device or any server device herein. Alternatively or inaddition, any method herein may be split between two or more devices,such as between a client device and a server device. Any database hereinmay be a relational database system, and may be Structured QueryLanguage (SQL) based.

Any method herein may comprise communicating, such as by the device,with an Internet-connected server. Any server herein may comprise adatabase, and any method herein may further comprise communicating withthe server for using or accessing the database. Any communicating hereinwith the server may be via a wireless network that may use a wirelesstransceiver and an antenna in the device. Any method herein may furthercomprise sending, via the wireless network to the server, the identifieditems or the estimated current location of the device.

Any method herein may use, or may be used with, a virtualization, andany communication herein, such as between the device and the server, maybe executed as a virtualized network as part of a Virtual Machine (VM).Any method herein may use, or may be used with, a host computer that mayimplement the VM, and any method herein may further comprise executing,by the host computer, a hypervisor or a Virtual Machine Monitor (VMM),and any virtualized network herein may use or may interface virtualhardware, and the virtualization may include, may be based on, or mayuse, full virtualization, para-virtualization, or hardware assistedvirtualization.

Any network herein may be a wireless network, the first port may be anantenna for transmitting and receiving first Radio-Frequency (RF)signals over the air, and the first transceiver may be a wirelesstransceiver coupled to the antenna for wirelessly transmitting andreceiving first data over the air using the wireless network.Alternatively or in addition, the network may be a wired network, thefirst port may be a connector for connecting to the network medium, andthe first transceiver may be a wired transceiver coupled to theconnector for transmitting and receiving first data over the wirelessmedium.

Any system, device, module, or circuit herein may be addressable in awireless network (such as the Internet) using a digital address that maybe a MAC layer address that may be MAC-48, EUI-48, or EUI-64 addresstype, or may be a layer 3 address and may be a static or dynamic IPaddress that may be of IPv4 or IPv6 type address. Any system, device, ormodule herein may be further configured as a wireless repeater, such asa WPAN, WLAN, or a WWAN repeater.

Any method herein, any step herein, any flow-chart herein, or any partthereof, may be used with a virtualization, and at least one of thesteps or methods herein may be executed as part of a virtualizedapplication as part of a Virtual Machine (VM). Any device herein, suchas the analyzer device, the first device, or any part thereof, may beimplemented as virtual hardware. Any virtualization herein may be usedwith an host computer that implement the VM, and may further comprisingexecuting, by the host computer, a hypervisor or a Virtual MachineMonitor (VMM). Any virtualized application herein or any or hardwarevirtualization herein may use or may interface virtual hardware. Anyvirtualization herein may include, may be based on, or may use, fullvirtualization, para-virtualization, or hardware assistedvirtualization.

Any operating system herein may be used with a virtualization, and anyoperating system herein may be executed as a guest operating system aspart of a Virtual Machine (VM). The virtualization may be implemented bya host computer that may implement the VM, and any method herein mayfurther comprise executing, by the host computer, a hypervisor or aVirtual Machine Monitor (VMM), and the guest operating system may use orinterface virtual hardware. Any such virtualization herein may include,may be based on, or may use, full virtualization, para-virtualization,or hardware assisted virtualization.

Any element or entity herein may be implemented as virtualized entity.Any virtualization may include, may be based on, or may use, desktopvirtualization, network virtualization, storage virtualization,application virtualization, server virtualization, or any combinationthereof. Further, any virtualization herein may include, may be basedon, or may use, full virtualization, para-virtualization, or hardwareassisted virtualization. Further, any virtualization herein may include,may be based on, or may use, a Virtual Machine (VM) on a host computerthat executes a hypervisor or Virtual Machine Monitor (VMM), and theoperating system may be a guest operating system that may use orinterface a virtual hardware.

Any method herein may be used with a virtualization, where at least oneof the steps may be executed as part of a virtualized application aspart of a Virtual Machine (VM). Alternatively or in addition, any deviceor server herein may be implemented as virtual hardware. Further, anymethod herein may be used with a host computer that may implement theVM, and any method herein may further comprise executing, by the hostcomputer, a hypervisor or a Virtual Machine Monitor (VMM), and anyvirtualized application herein or any hardware herein may use or mayinterface virtual hardware. Any virtualization herein may include, maybe based on, or may uses, full virtualization, para-virtualization, orhardware assisted virtualization. At least two devices may bevirtualized by the same host computer that implements the VM.

Any selection herein, such as any selection from any list or group, maybe based on, or may use, a load balancing, a First-In-First-Out (FIFO)scheme, a Last-In-First-Out (LIFO) scheme, a sequential or cyclicselection, a random selection, or any combination thereof. Any randomselection herein may use, or may be based on, one or more random numbersgenerated by a random number generator. Any random number generatorherein may be hardware based, and may be using thermal noise, shotnoise, nuclear decaying radiation, photoelectric effect, or quantumphenomena. Alternatively or in addition, any random number generatorherein may be software based, and may be based on executing an algorithmfor generating pseudo-random numbers.

The above summary is not an exhaustive list of all aspects of thesystems and methods described. Indeed, it is contemplated that thisdocument covers all systems and methods that can be practiced from allsuitable combinations and derivatives of the various aspects summarizedabove, as well as those disclosed in the detailed description below, andparticularly pointed out in the claims filed with the application. Suchcombinations have particular advantages not specifically recited in theabove summary.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the system and method are herein described, by way ofnon-limiting examples only, with reference to the accompanying drawings,wherein like designations denote like elements. Understanding that thesedrawings only provide information concerning typical embodiments of theinvention and are not therefore to be considered limiting in scope:

FIG. 1 illustrates schematically a block diagram of a prior-art computerconnected to the Internet.

FIG. 1 a illustrates schematically prior-art servers, clients, and acomputer workstation connected via the Internet.

FIG. 2 illustrates schematically a prior-art arrangement ofvirtualization.

FIG. 2 a illustrates schematically a prior-art arrangement of hostedarchitecture of virtualization.

FIG. 2 b illustrates schematically a prior-art arrangement of bare-metal(hypervisor) architecture of virtualization.

FIG. 3 illustrates schematically a block diagram of an arrangement thatincludes a server that communicates over the Internet with a clientdevice such as a smartphone or tablet.

FIG. 4 schematically illustrates a schematic diagram of an example of afeed-forward Artificial Neural Network (ANN).

FIG. 4 a schematically illustrates a schematic diagram of examples of aDeep Neural Network (DNN).

FIG. 4 b depicts schematically an HMD with and without antennas.

FIG. 4 c depicts schematically a person wearing an HMD.

FIG. 5 depicts schematically general views of various stringinstruments.

FIG. 5 a depicts schematically general views of various woodwindinstruments.

FIG. 5 b depicts schematically general views of various brassinstruments.

FIG. 5 c depicts schematically general views of various percussioninstruments.

FIG. 6 depicts schematically examples of popular musical symbolsaccording to a common music notation convention.

FIG. 6 a depicts schematically a correspondence of musical symbols andthe associated piano keys.

FIG. 6 b depicts schematically a sheet music of a popular song.

FIG. 6 c depicts schematically an arrangement of a prior art onlinepiano playing learning session using a tablet.

FIG. 7 illustrates a simplified schematic flow chart of a method forestimating and modifying complexity or difficulty of a sheet music.

FIG. 8 depicts schematically an arrangement of an online piano playinglearning using a client device and a server device.

FIG. 8 a depicts schematically an arrangement of an online piano playinglearning with the data paths between the client device and the serverdevice.

FIG. 8 b depicts schematically an arrangement of an online piano playinglearning with databases that relate to users and to musical pieces.

FIG. 9 illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on a delay in the serverdevice.

FIG. 10 illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on a delay in the clientdevice.

FIG. 11 illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on sequentially displayingand playing parts of a musical piece.

FIG. 11 a illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on sequentially displayingplaying parts of a musical piece and on audible instructions or cues.

FIG. 12 illustrates a simplified schematic flow chart of a method forhandling errors as part of an adaptive and interactive music learning.

FIG. 13 illustrates a simplified schematic flow chart of a method forcounting errors and acting according to the count obtained duringpracticing of a musical piece as part of an adaptive and interactivemusic learning.

FIG. 13 a illustrates a simplified schematic flow chart of a method forcounting errors and acting according to the count obtained duringpracticing of part of a musical piece as part of an adaptive andinteractive music learning.

FIG. 13 b illustrates a simplified schematic flow chart of a method forno delay practicing and for counting errors and acting according to themultiple complexity- or features-based count obtained during practicingof part of a musical piece as part of an adaptive and interactive musiclearning.

FIG. 14 depicts schematically musical pieces database that includesmultiple versions or variants for one or more of the musical pieces thatcorresponds to various complexity or difficulty levels.

FIG. 14 a illustrates a simplified schematic flow chart of a method foradapting a simplified version of a selected musical piece to a personskill level.

FIG. 14 b illustrates a simplified schematic flow chart of a method foradapting a simplified version of part of a selected musical piece to aperson skill level during playing using stored simplified version.

FIG. 14 c illustrates a simplified schematic flow chart of a method forcreating a simplified version of a selected musical piece according to aperson skill level in real-time during the playing session.

FIG. 14 d illustrates a simplified schematic flow chart of a method forcreating a simplified version of a selected part of a selected musicalpiece according to a person skill level in real-time during the playingsession.

FIG. 15 depicts schematically an arrangement of an online piano playinglearning using a client device.

FIG. 15 a illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on sequentially playingparts of a musical piece using a client device.

FIG. 15 b depicts schematically an arrangement of a MIDI controllerwiredly connected to a client device.

FIG. 15 c depicts schematically an arrangement of a MIDI controllerwirelessly connected to a client device that is connected forcloud-based service.

FIG. 15 d illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning using a MIDI controller based ona delay in the server device.

FIG. 16 illustrates a simplified schematic table that associates to amusical piece various musical instruments and instructions for playingthe musical piece.

FIG. 16 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece.

FIG. 17 illustrates a simplified schematic flow chart of a method forcooperatively practicing a musical piece by a musical instrument whilehearing a background of additional musical instruments, where at leastone of the additional musical instruments is artificially produced basedon the vocalizing musical symbols.

FIG. 17 a illustrates a simplified schematic flow chart of a method forcooperatively practicing a musical piece by a musical instrument whilehearing a background of additional musical instruments, where at leastone of the additional musical instruments is artificially produced basedon stored or received sound files.

FIG. 18 depicts schematically an arrangement of three users practicing amusical piece at a same location using three different musicalinstruments.

FIG. 18 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece in a scenario of playing at the same location.

FIG. 19 depicts schematically an arrangement of three users practicing amusical piece at three different locations using three different musicalinstruments.

FIG. 19 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece in a scenario of playing at differentlocations, without cooperation between the users.

FIG. 20 depicts schematically an arrangement of three users practicing amusical piece at three different locations using three different musicalinstruments and using artificially produced sound of various musicalinstruments.

FIG. 20 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece in a scenario of playing at differentlocations, while using artificially produced sound of various musicalinstruments.

FIG. 21 depicts schematically an arrangement of three users practicing amusical piece at three different locations using three different musicalinstruments, where two users hear the playing of other musicalinstruments by other users.

FIG. 21 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece in a scenario of playing at differentlocations, where two users hear the playing of other musical instrumentsby other users.

FIG. 22 depicts schematically an arrangement of three users practicing amusical piece at three different locations using three different musicalinstruments, where all the users hear the playing of other musicalinstruments by all other users.

FIG. 22 a illustrates a simplified schematic practice tables thatassociate, for each user, a source of each musical instrument that ispart of the musical piece in a scenario of playing at differentlocations, where all the users hear the playing of other musicalinstruments by all other users.

FIG. 23 depicts schematically an example of an architecture of softwaremodules for teaching of practicing of a musical instrument that adaptsto the user skill level;

FIG. 24 illustrates a simplified schematic flow chart of a method foradaptive and interactive music learning based on sequentially playingparts of a musical piece using a client device overlayed with example ofan architecture of associated software modules.

FIG. 25 is a schematic illustration of a user and the entities involvedin a music learning system, in accordance with some embodiments of thedisclosure.

FIG. 26 is a flowchart of steps in a method for teaching playing amusical instrument, in accordance with some embodiments of thedisclosure.

FIG. 27 is a flowchart of steps in a method for selecting andconfiguring I/O devices to be used when teaching a user to play, inaccordance with some embodiments of the disclosure.

FIG. 28 is a detailed flowchart of steps in a method for selecting andconfiguring I/O devices to be used when teaching a user to play, inaccordance with some embodiments of the disclosure.

FIG. 29 is a block diagram of a system for selecting I/O devices whenteaching a user to play, in accordance with some embodiments of thedisclosure.

DETAILED DESCRIPTION

The principles and operation of an apparatus according to the presentinvention may be understood with reference to the figures and theaccompanying description wherein similar components appearing indifferent figures are denoted by identical reference numerals. Thedrawings and descriptions are conceptual only. In actual practice, asingle component can implement one or more functions. Alternatively orin addition, each function can be implemented by a plurality ofcomponents and devices. In the figures and descriptions, identicalreference numerals indicate those components that are common todifferent embodiments or configurations. Identical numerical references(even in the case of using different suffix, such as 5, 5 a, 5 b and 5c) refer to functions or actual devices that are either identical,substantially similar, or having similar functionality. It will bereadily understood that the components of the present invention, asgenerally described and illustrated in the figures herein, could bearranged and designed in a wide variety of different configurations.Thus, the following more detailed description of the embodiments of theapparatus, system, and method of the present invention, as representedin the figures herein, is not intended to limit the scope of theinvention, as claimed, but is merely the representative embodiments ofthe invention. It is to be understood that the singular forms “a,” “an,”and “the” herein include plural referents unless the context clearlydictates otherwise. Thus, for example, reference to “a componentsurface” includes reference to one or more of such surfaces. By the term“substantially” it is meant that the recited characteristic, parameter,or value need not be achieved exactly, but that deviations orvariations, including for example, tolerances, measurement error,measurement accuracy limitations and other factors known to those ofskill in the art, may occur in amounts that do not preclude the effectthe characteristic was intended to provide.

Although the terms first, second, third, etc. may be used herein todescribe various elements, components, regions, layers and/or sections,these elements, components, regions, layers and/or sections should notbe limited by these terms. These terms may be only used to distinguishone element, component, region, layer or section from another region,layer or section. Terms such as “first,” “second,” and other numericalterms when used herein do not imply a sequence or order unless clearlyindicated by the context. Thus, a first element, component, region,layer or section discussed below could be termed a second element,component, region, layer or section without departing from the teachingsof the example embodiments.

Spatially relative terms, such as “inner”, “outer”, “beneath”, “below”,“right”, left”, “upper”, “lower”, “above”, “front”, “rear”, “left”,“right” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. Spatially relative terms maybe intended to encompass different orientations of the device in use oroperation in addition to the orientation depicted in the figures. Forexample, if the device in the figures is turned over, elements describedas “below” or “beneath” other elements or features would then beoriented “above” the other elements or features. Thus, the example term“below” can encompass both an orientation of above and below. The devicemay be otherwise oriented (rotated 90 degrees or at other orientations)and the spatially relative descriptors used herein interpretedaccordingly.

In one example, it may be beneficial to adapt a music learning practicesession to the specific user skill or ability. For example, a sheetmusic may be transcribed to reduce complexity of difficulty, eitheroff-line according to estimated or expected user playing ability, or ‘onthe fly’ in real-time during the practice session, to better adapt tothe student skill level, such as by executing a flow chart 70 shown inFIG. 7 . A sequence of symbols that corresponds to playing of a musicalpiece on a musical instrument, such as in a sheet music form, isobtained as part of an “Obtain Sheet Music” step 71. The sequence ofsymbols may be retrieved from a local memory, or from a storage of aremote server over the Internet. In case where the sequence of symbols(such as the sheet music) is received from a remote location as part ofthe “Obtain Sheet Music” step 71, it may be locally stored for use aspart of a “Store Version ‘5’” step 74.

The obtained sequence, referred to as the ‘original’ sequence or version‘5’, is analyzed for estimating the playing complexity or difficulty aspart of an “Estimate Complexity” step 72. The estimated (or calculated)complexity or difficulty level is stored as associated with the obtainedsequence of symbols of the musical piece as part of a “Store Level” step77.

The estimation of the playing complexity or difficulty in the “EstimateComplexity” step 72 may be based on, may use, or may be according to,computationally quantifying performance difficulty as well as musicalfidelity to the original score, and formulating the problem asoptimization of musical fidelity under constraints on difficulty valuesusing a statistical-modeling as presented in the article entitled:“Statistical piano reduction controlling performance difficulty” by EitaNakamura and Kazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published2018 in SIP (2018), vol. 7, e13; analyzing the original music in orderto determine the type of arrangement element performed by an instrument,then identifying each phrase and associated it with a weightedimportance value as described in the article entitled: “Automatic Systemfor the Arrangement of Piano Reductions” by Shih-Chuan Chiu, Man-KwanShan, and Jiun-Long Huang, published 2009 in the 11th IEEE InternationalSymposium on Multimedia; analyzing the semantic content of music byfocusing on symbolic music, i.e., sheet music or score, formulatingdifficulty level recognition as a regression problem to predict thedifficulty level of piano sheet music, and recognizing the difficultylevel of piano sheet music as described in the article entitled: “AStudy on Difficulty Level Recognition of Piano Sheet Music” byShih-Chuan Chiu and Min-Syan Chen, published December 2012 in the ISM'12: Proceedings of the 2012 IEEE International Symposium on Multimedia;Using Score Analyzer that is based on seven criteria to characterizetechnical instrumental difficulties in order to automatically extractthe difficulty level of a MusicXML piece, and suggest advice based on aMusical Sign Base (MSB), as proposed in the article entitled: “SCOREANALYZER: AUTOMATICALLY DETERMINING SCORES DIFFICULTY LEVEL FORINSTRUMENTAL E-LEARNING” by Véronique Sébastien, Henri Ralambondrainy,Olivier Sébastien, and Noel Conruyt of IREMIA—Laboratoire d'Informatiqueet de Mathématiques, EA2525 University of Reunion Island, Saint-Denis,Reunion (FRANCE), published October 2012 in 13th International Societyfor Music Information Retrieval Conference (ISMIR 2012); using asystematic and objective approach to computational assessment of thecomplexity of a music score for any instrument, as introduced in thepaper entitled: “Musiplectics: Computational Assessment of theComplexity of Music Scores” by Ethan Holder, Eli Tilevich, and AmyGillick, published October 2015 in ONWARD '15 [ACM978-1-4503-1995-9/13/10, http://dx.doi.org/10.1145/2508075.2514879], ordetermining the degree of difficulty by converting the score, which isexpressed as a traditional music score, into electronic music sheet andcalculating information about the elements needed to play sheet music bydistance of notes, tempo, and quantifying the ease of interpretation, asproposed in the paper entitled: “A Method for Measuring the Difficultyof Music Scores” by Yang-Eui Song and Yong Kyu Lee [www.ksci.re.krhttp://dx.doi.org/10.9708/jksci.2016.21.4.039] published April 2016 inthe Journal of The Korea Society of Computer and Information Vol. 21 No.4.

Alternatively or in addition, the estimation of the playing complexityor difficulty in the “Estimate Complexity” step 72 may be based on, mayuse, or may be according to, extracting features of the obtained sheetmusic as part of a “Extract Features” step 75, and calculatingdifficulty characteristics on the mapped features as part of a“Calculate Difficulty” step 76. The extracted features as part of the“Extract Features” step 75 may include notes or chords difficulty,pertaining to how the player fingers are required to be positioned,transition between notes/chords difficulty, defined Tempo, or associateddecorations, such as excessive playing time or loudness.

In one example, the “Calculate Difficulty” step 76 may be based on, mayuse, or may be include, difficulty or skill characteristics, such asrhythmic difficulty, relating to how irregularly musical notes andchords are located in time, and what is the “speed” of music (the rateat which notes/chords should be played); motoric difficulty, relating tohow difficult it is physically to produce the musical sounds using amusical instrument (usually mainly related to the complexity of thephysical arrangement of user's fingers); harmonic difficulty, relatingto how complex combinations of musical notes need to be playedsimultaneously (as chords) or sequentially (as very unpredictablemelodies); or expressivity difficulty, relating to the user need tocontrol and vary the loudness (“dynamics”) and timbre (“tone color”) ofthe musical sounds being produced. In one example, such analysis may bebased on the U.S. Pat. No. 9,767,705 to Klapuri et al. entitled: “Systemfor estimating user's skill in playing a music instrument anddetermining virtual exercises thereof”, which is incorporated in itsentirety for all purposes as if fully set forth herein.

In one example, multiple levels of complexity or difficulty are defined,where Level ‘5’ corresponds to the most complex or difficult playingrequirement, which is typically associated with the original arrangementthat is obtained in the “Obtain Sheet Music” step 71. In some cases, thecomplexity or difficulty estimated as part of the Estimate Complexity”step 72 may be appropriate for playing by an experienced player, but notsuitable to be played by a beginner or student. In such a case, it maybe beneficial to transcribe the sequence of musical symbols (or sheetmusic) to provide lower complexity or less difficult sequence ofsymbols, while retaining the general character or feeling of theoriginal musical piece. In one example, four versions of sequences ofmusical symbols, referred to as versions 4, 3, 2, and ‘1’, aregenerated, which are all transcribed or derived of the original version‘5’. For example, version ‘4’ may be slightly simplified from theoriginal version ‘5’, thus suited to more experienced players, rangingto version ‘1’, that is the most simplified version, which may besuitable to be played by novice players or beginners. While fourversions are exampled, any number of transcribed versions may equally beproduced, such as 1, 2, 3, 5, 7, 10, 15, 20, or more versions. The fourversions are generated as part of a “Modify Arrangement” step 73, andthen may be stored for future use, such as storing version ‘4’ as partof a “Store Version ‘4’” step 74 a, storing version ‘3’ as part of a“Store Version ‘3’” step 74 b, storing version ‘2’ as part of a “StoreVersion ‘2’” step 74 c, and storing version ‘1’ as part of a “StoreVersion ‘1’” step 74 d.

The transcribing or modifying of the original version ‘5’ for producingsimplified versions as part of “Modify Arrangement” step 73, may bebased on, may use, or may be according to, the article entitled:“Statistical piano reduction controlling performance difficulty” by EitaNakamura and Kazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published2018 in SIP (2018), vol. 7, e13; the article entitled: “Automatic Systemfor the Arrangement of Piano Reductions” by Shih-Chuan Chiu, Man-KwanShan, and Jiun-Long Huang, published 2009 in the 11th IEEE InternationalSymposium on Multimedia; the U.S. Pat. No. 9,767,705 to Klapuri et al.entitled: “System for estimating user's skill in playing a musicinstrument and determining virtual exercises thereof”; or by availablecommercial application such as Ludwig (www.write-music.com/) andAnthemScore (www.lunaverus.com/).

An example of a system for online music learning by the person 36 ofplaying a musical instrument, such as the piano 83, that is based on theclient device 35 that communicates with a server device 23 a over theInternet 22, is shown in an arrangement 80 shown in FIG. 8 , in anarrangement 80 a shown in FIG. 8 a , and in an arrangement 80 b shown inFIG. 8 b . The client device 35 may be identical to, similar to, partof, comprise, or integrated with, the computer system 10 shown in FIG. 1or the client device 35 shown in FIG. 3 . In one example, the clientdevice 35, which may be a smartphone or a tablet computer, is usedmainly for the purpose of interacting with the user 36 and the piano 83,while the processing is mainly performed in the server device 23 a. Theclient device communicates with the server 23 a over the Internet 22using a wireless communication that involves the wireless transceiver 28connected to the antenna 29. As shown in the arrangement 80 a in FIG. 8a , the teaching scheme is based on the server device 23 a sendingvisual data over a path 79 a (shown in FIG. 8 a ) that includes theInternet 22, to be displayed to the user 36 via a display 81 in theclient device 35. The sound generated by the piano 83 is captured by amicrophone 82, where it is converted to a digital representation andsent to the server device 23 a over the Internet 22 over a path 79 b(shown in FIG. 8 a ), to be further processed therein.

The arrangement 80 shown in FIG. 8 , or the client device 35, may beused with, integrated with, or used in combination with, a VirtualReality (VR) system simulating a virtual environment to a person, andmay be used by the VR system. The communication with the VR system maybe wired or wireless, and the VR system may comprise a Head-MountedDisplay (HMD).

The display 81 may be identical to, similar to, part of, comprise, orintegrated with, the display 17 shown in FIG. 1 , and may be identicalto, similar to, part of, comprise, or integrated with, the outputcomponent 34 of the arrangement 30 in FIG. 3 . The display 81 mayconsist of, or may comprise, a display screen for visually presentinginformation to the user 36, and the display or the display screen mayconsist of, or may comprise, a monochrome, grayscale or color display,having an array of light emitters or light reflectors.

Alternatively or in addition, the display 81 or the display screen mayconsist of, or may comprise, an analog display having an analog inputinterface that supports NTSC, PAL or SECAM formats, and the analog inputinterface may comprise of RGB, VGA (Video Graphics Array), SVGA (SuperVideo Graphics Array), SCART or S-video interface. Furthermore, thedisplay 81 or the display screen may consist of, or may comprise, adigital display having a digital input interface that is IEEE1394,FireWire™, USB, SDI (Serial Digital Interface), HDMI (High-DefinitionMultimedia Interface), DVI (Digital Visual Interface), UDI (UnifiedDisplay Interface), DisplayPort, Digital Component Video, or DVB(Digital Video Broadcast) interface.

For example, the display 81 or the display screen may be based on, oruse, a Cathode-Ray Tube (CRT) display, a Field Emission Display (FED),an Electroluminescent Display (ELD), a Vacuum Fluorescent Display (VFD),an Organic Light-Emitting Diode (OLED) display, a passive-matrix(PMOLED) display, an active-matrix OLEDs (AMOLED) display, a LiquidCrystal Display (LCD) display, a Thin Film Transistor (TFT) display, anLED-backlit LCD display, or an Electronic Paper Display (EPD) displaythat is based on Gyricon technology, Electro-Wetting Display (EWD), orElectrofluidic display technology. Further, the display 81 or thedisplay screen may consist of, or may comprise, a laser video displaythat is based on a Vertical-External-Cavity Surface-Emitting-Laser(VECSEL) or a Vertical-Cavity Surface-Emitting Laser (VCSEL).Furthermore, the display 81 or the display screen may consist of, or maycomprise, a segment display based on a seven-segment display, afourteen-segment display, a sixteen-segment display, or a dot matrixdisplay, and may be operative to only display at least one of digits,alphanumeric characters, words, characters, arrows, symbols, ASCII, andnon-ASCII characters.

The microphone 82 may be identical to, similar to, part of, comprise, orintegrated with, the input component 38 of the arrangement 30 in FIG. 3. The microphone 82 is configured to responds to audible or inaudiblesound, and is an omnidirectional, unidirectional, or bidirectionalmicrophone. The microphone 82 may be based on the sensing the incidentsound-based motion of a diaphragm or a ribbon, or may consist of, or maycomprise, a condenser, an electret, a dynamic, a ribbon, a carbon, anoptical microphone, or a piezoelectric microphone. Further, themicrophone 82 may consist of, may comprise, or may be based on, amicrophone array for improving directivity.

The client device 35 may further comprise a sounder, such as a speaker78, that may be identical to, similar to, part of, comprise, orintegrated with, the output component 34 of the arrangement 30 in FIG. 3. The sound by any sounder such as the speaker 78 may be audible orinaudible (or both), and may be omnidirectional, unidirectional,bidirectional, or provide other directionality or polar patterns. Thespeaker 78 may be an electromagnetic loudspeaker, a piezoelectricspeaker, an Electro-Static Loudspeaker (ESL), a ribbon or planarmagnetic loudspeaker, or a bending wave loudspeaker. The sounder, suchas the speaker 78, may consist of, may comprise, may use, or may bebased on, an electric sound source that may convert electrical energyinto sound waves, and the electric sound source may be configured toemit an audible or inaudible sound using omnidirectional,unidirectional, or bidirectional pattern. Further, the sounder, such asthe speaker 78, may convert electrical energy to sound waves transmittedthrough the air, an elastic solid material, or a liquid, usually bymeans of a vibrating or moving ribbon or diaphragm.

Further, the sounder, such as the speaker 78, may comprise, may use, ormay be based on, an electromagnetic loudspeaker, a piezoelectricspeaker, an electrostatic loudspeaker (ESL), a ribbon magneticloudspeaker, a planar magnetic loudspeaker, or a bending waveloudspeaker. Furthermore, the sounder may consist of, may comprise, mayuse, or may be based on, an electromechanical scheme or a ceramic-basedpiezoelectric effect. In addition, the sounder may consist of, maycomprise, may use, or may be based on, an ultrasonic transducer that maybe a piezoelectric transducer, crystal-based transducer, a capacitivetransducer, or a magnetostrictive transducer.

While piano 83 is exampled in FIG. 8 , any musical instrument may beequally used. For example, the musical instrument may consist of, or maycomprise, a Soprano instrument such as a flute, a violin, a sopranosaxophone, a trumpet, a clarinet, an oboe, or a piccolo. Alternativelyor in addition, any musical instrument herein may consists of, or maycomprise, an Alto instrument, such as an alto saxophone, an French horn,an English horn, a viola, or an alto horn. Alternatively or in addition,any musical instrument herein may consists of, or may comprise, a Tenorinstrument, such as a trombone, a tenoroon, a tenor saxophone, a tenorviolin, a guitar, or a tenor drum. Alternatively or in addition, anymusical instrument herein may consists of, or may comprise, a Baritoneinstrument, such as a bassoon, a baritone a saxophone, a bass clarinet,a cello, a baritone horn, or an euphonium. Alternatively or in addition,any musical instrument herein may consists of, or may comprise, a Bassinstrument, such as a double bass, a bass guitar, a contrabassoon, abass saxophone, a tuba, or a bass drum.

Further, the musical instrument may consist of, or may comprise, astring instrument that produces sound by means of vibrating strings,such as a guitar, an electric bass, a violin, a viola, a cello, a doublebass, a banjo, a mandolin, an ukulele, or a harp. Any string instrumentherein may consist of, or may comprise, a lute instrument in which thestrings are supported by a neck and a bout, a harp instrument, in whichthe strings are contained within a frame, or a zither instrument, inwhich the strings are mounted on a body. Further, any string instrumentherein be configured to be played by plucking, bowing, or striking.Furthermore, the string instrument any string instrument herein may bean acoustic instrument, or alternatively may comprise an electricamplification.

Furthermore, the musical instrument may consist of, or may comprise, awoodwind instrument that produces sound when the player blows airagainst a sharp edge or through a reed, causing the air within itsresonator to vibrate, such as a flute, a piccolo, an oboe, an Englishhorn, a clarinet, a bass clarinet, a bassoon, a contrabassoon, or aSaxophone. Any woodwind instrument herein may consist of, or maycomprise, a flute that is configured to produce sound when air is blownacross an edge, and any flute herein may consist of, or may comprise, anopen flute, in which the player's lips form a stream of air which goesdirectly from the players lips to the edge, or a closed flute, in whichin which the musical instrument has a channel to form and direct the airstream over an edge. Alternatively or in addition, any woodwindinstrument herein may consist of, or may comprise, a reed instrumentthat is configured to produce sound by focusing air into a mouthpiecewhich then causes a reed, or reeds, to vibrate, and any reed instrumentherein may consist of, or may comprise, a single reed instrument thatuses a reed, which is a thin-cut piece of cane or plastic that is heldagainst the aperture of a mouthpiece with a ligature, so that when airis forced between the reed and the mouthpiece, the reed vibrates tocreate a sound, a double reed instrument that uses two precisely cutsmall pieces of cane that are joined together at the base and areinserted into the top of the instrument to vibrate as air is forcedbetween the two pieces, or a capped double reed instrument configured tovibrate when the player blows through a hole in a cap that covers thereed.

In addition, the musical instrument may consist of, or may comprise, abrass instrument that produces sound by sympathetic vibration of air ina tubular resonator in sympathy with the vibration of the player's lips,such as a Trumpet, a Cornet, a Horn, a Trombone, a Saxhorn, or a Tuba.Any brass instrument herein may consist of, or may comprise, a valvedbrass instrument that uses a set of valves operated by the player'sfingers that introduce additional tubing, or crooks, into the instrumentfor changing its overall length, or a slide brass instrument that uses aslide to change the length of tubing.

Even more, the musical instrument may consist of, or may comprise, apercussion instrument that is configured to produce sound by beingstruck or scraped by a beater, such as a timpani, a snare drum, a bassdrum, cymbals, a triangle, a tambourine, a glockenspiel, or a xylophone.Any percussion instrument herein may consist of, or may comprise, apitched percussion instrument that produces notes with an identifiablepitch, or an unpitched percussion instrument that produces notes orsounds in an indefinite pitch.

Alternatively or in addition, the musical instrument may comprise, ormay be based on, an acoustic, stringed musical instrument, configured sothat the strings are struck by wooden hammers that are coated with asofter material, and the musical instrument may further comprise akeyboard that may consist of a row of keys, may be configured so thatthe performer presses down or strikes with the fingers and thumbs ofboth hands to cause the hammers to strike the strings. Any musicalinstrument herein may comprise, or may consist of, a piano, such as thepiano 83, which may comprise, or may consist of, a grand piano, anupright piano, or an electronic piano.

Further, any playing of any musical instrument herein may consists of,may comprises, or may be supplemented with, a vocal music, that may bewith or without instrumental accompaniment, and may use lyrics, such assinging.

The display 81 in the client device 35 notifies the person 36 how toplay the piano 83 by displaying a sequence 85 of musical symbols, whichmay be a simple training set for acquiring or enhancing the playingskills, or which may be according to musical notation standard orconvention. For example, the sequence 85 shown on the display 81 may bepart of, or an entire of, a musical piece. Further, the sequence 85shown on the display 81 may be adapted to the specific played musicalinstrument, such as specially adapted to the piano 83. In one example,the sequence 85 shown on the display 81 is a part of, or an entire of, asheet music of a musical piece, and each of the musical symbols in thesequence may be according to a musical notation convention and typicallyincludes lines, clefs, notes, and rests, and may notate pitch, tempo,meter, duration, or articulation of a note or a passage of music, andwherein the sequence.

The sequence may include one or more clefs that defines the pitch rangeor the tessitura of the symbol on which it is placed, one or moremusical notes that denote a musical sound or the pitch and duration of amusical sound, one or more accidentals that denote a pitch or a pitchclass, one or more key signatures that define the prevailing key of themusic that follows, one or more time signatures that defines the meterof the music, one or more lines or a note relationships, one or moredynamics that indicate the relative intensity or volume of a musicalline, one or more articulations that specify how to perform individualnotes within a phrase or passage, one or more ornaments that modify thepitch pattern of an individual note, one or more octave signs,repetitions or codas, one or more pedal marks, or any combinationthereof.

As part of the teaching session, the user 36 is expected to follow thedisplayed sequence 85 of the musical symbols, and to timely andaccurately play by pressing the piano 83 keys (and pedals) in responseto the shown symbols. In one example, a single symbol is shown at atime, such as the symbol 85 b shown in FIG. 8 a . Alternatively oraddition, multiple symbols 85 a are displayed, such as the entire sheetmusic, or a part thereof of that includes one or more lines. In such acase, the symbol that is to be next played by the user 36 isspecifically notified or marked on the display 81, such as by changingcolor, enclosing it in a frame (such as a rectangular shape), or bypointing at it using an arrow 85 c, in order to clearly notify theplayer 36 what is the next symbol to be played. Preferably, the pace ortempo of displaying of the symbol in the sequence 85 are based on theskill level of the user 36, on the tempo or pace associated with themusical piece that is represented by the sequence 85 (such as the tempothat is set or recommended by the creator or composer of the piece, orby a performer), or a combination thereof. For example, in case wherethe sequence 85 is associated with a musical piece that defines aspecific tempo, the actual pace of displaying the symbols to the user 36may be less than 95%, 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%,40%, 35%, 30%, 25%, 20%, 15%, or 10% of the defined tempo or pace, ormay be at least 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%,35%, 30%, 25%, 20%, 15%, 10%, or 5% of the defined tempo or pace.

The server device 23 a may comprise, may communicate with, or may beconnected to, a memory 31 that may store one or more databases, such asfor managing and handling a service of online music learning. The memory31 may store a user's database 31 a that associates skill level valuesto various respective users, such as the user 36, as shown in a table 86(or in any other data structure) in FIG. 8 b . The skill level valuesmay directly or indirectly map to the complexity or difficulty levelthat is calculated or estimated in the “Estimate Complexity” step 72 inthe flow chart 70. For example, a user value of ‘5’ corresponds to anexpert that can play all musical pieces, a user value of ‘4’ correspondsto an less experienced user that is skilled enough to play all musicalpieces associated with complexity or difficulty level of ‘4’ or less,and a user value of ‘1’ corresponds to a beginner or novice user thatcan properly play only musical pieces associated with complexity ordifficulty level of ‘1’.

A top row 84 indicates that a left column 86 a includes identifiers ofusers, and a right column 86 b associates a skill level value to each ofthe users. In the example of the table 86, a user identified as a User#1 84 a is associated with a skill level value of ‘1’, a user identifiedas an User #2 84 b is associated with a skill level value of ‘3’, a useridentified as an User #3 84 c is associated with a skill level value of‘2’, and a user identified as an User #4 84 d is associated with skilllevel value of ‘5’. While only exampled for four users, any number ofusers may be equally used.

The identifiers of the user shown in the column 86 a may typicallycomprise identifiers of the persons, such as the user 36, who are using,or are expected to use, the system 80. Such identifiers may compriseactual names, identification numbers (such as government identifyingnumbers), usernames, login names, e-mail address, telephone number,screennames (or screen names), account names, or nicknames, or any otheridentifiers of end users, which are the ultimate human users thatoperates, own, or use, the client device 35. Alternatively or inaddition, the user may choose to be anonymous, and will not provide anyidentifying details.

For example, a skill level value of ‘1’ may refer to a novice student,having no experience with music or with playing of the respectivemusical instrument, such as the piano 83, and thus is expected to makemany errors while playing. Such a user requires the slowest pace ortempo, and/or simplified music arrangement, to keep up with the sequence85 shown on the display 81. In contrast, a skill level value of ‘5’ mayrefer to an experienced or competent level student, having substantialexperience with playing of the respective musical instrument, such asthe piano 83, and thus is expected to make no or minimal number oferrors while playing. Such a user may easily track and play the fastestpace or tempo (or the defined tempo of a respective musical piece)and/or more complex music arrangement, and is easily able to keep upwith the sequence 85 shown on the display 81. A skill level value of ‘3’may refer to a medium level student, having some experience with playingof the respective musical instrument, such as the piano 83, and thus isexpected to make reasonable number of errors while playing. Such a userrequires the somewhat lower pace or tempo of the maximum or definedtempo, and/or medium level music arrangement, to keep up with thesequence 85 shown on the display 81.

The memory 31 may further store a pieces-database 31 b that may storesheet music for various musical pieces, together with respectivemetadata such as name or another identifier of the piece, type of thepiece, the tempo associated with, or defined to, the piece, as shown ina table 8 b (or in any other data structure) in FIG. 89 .

A top row 88 identifies a first left column 87 a as ‘Name/ID’, referringto an identifier of the musical piece, such as a name or a numeralidentifier, a second column 87 b as ‘Type’ referring to type or genre ofthe musical piece, a third column 87 c as ‘Tempo’, referring to defined,suggested, or recommended tempo associated with the respective musicalpiece, and a fourth column 87 d as ‘File’, identifying the filename thatholds the respective sheet music (or other musical symbols sequence) ofthe respective musical piece. The complexity or difficulty level of therespective musical piece 88, which may be estimated or calculated aspart of the “estimate Complexity” step 72 and may be stored in the table86 as part of the “Store level” step 77 shown in the flow chart 70 inFIG. 7 , are shown in a fifth column 87 e. The table 89 examples Nrecords of musical pieces, and N can be any number. In one example, thefile may be a MIDI file or Music XML format.

In the example of the table 89, a musical piece 88 a is identified by aname (or another identifier) Piece #1, the type of the piece is “Aria”,the associated tempo is “Largo”, the sheet music file is namedsheet1.xml, denoting a MusicXML file format, and the correspondingcomplexity or difficulty level is ‘2’. Similarly, a musical piece 88 bis identified by a name (or another identifier) Piece #2, the type ofthe piece is “Song”, the associated tempo is “Allegro”, the sheet musicfile is named sheet2.pdf, and the corresponding complexity or difficultylevel is ‘4’, a musical piece 88 c is identified by a name (or anotheridentifier) Piece #3, the type of the piece is “Song”, the associatedtempo is “Vivace”, the sheet music file is named sheet3.doc, and thecorresponding complexity or difficulty level is ‘3’, a musical piece 88d is identified by a name (or another identifier) Piece #4, the type ofthe piece is “Poem”, the associated tempo is “Presto”, the sheet musicfile is named sheet4.pdf, and the corresponding complexity or difficultylevel is ‘5’, and a musical piece 88N is identified by a name (oranother identifier) Piece #N, the type of the piece is “Sonata”, theassociated tempo is “Adagio”, the sheet music file is named sheetN.mid(denoting a MIDI file format), and the corresponding complexity ordifficulty level is ‘4’. In one example, another element of metadata maybe to associate a respective skill level value. For example, in case askill level value ‘3’ is associated with a musical piece, it suggestthat a user 36 with skill level of ‘3’, such as the User #2 84 b shownas part of the table 86, is adapted to reasonably play this difficultylevel. While exampled using 'pdf, .doc and .docx documents, any otherformat of a file or document may be equally used.

The tempo in the column 87 c may be based on, may consist of, or maycomprise, Larghissimo, Adagissimo, Grave, Largo, Lento, Larghetto,Adagio, Adagietto, Andante, Andantino, Marcia moderato, Andantemoderato, Moderato, Allegretto, Allegro moderato, Allegro, MoltoAllegro, Vivace, Vivacissimo, Allegrissimo, Allegro vivace, Presto, orPrestissimo. Alternatively or in addition, the tempo or pace in thecolumn 87 c may be equally represented in beats per minute (bpm), suchas above 24 bpm, above 200 bpm, or within one of the ranges of 25-45bpm, 40-60 bpm, 45-60 bpm, 60-66 bpm, 66-76 bpm, 72-76 bpm, 70-80 bpm,76-108 bpm, 80-108 bpm, 83-85 bpm, 92-98 bpm, 98-112 bpm, 102-110 bpm,116-120 bpm, 120-156 bpm, 124-156 bpm, 156-176 bpm, 172-176 bpm, or168-200 bpm.

Each of the sequences of musical symbols 88 a-88N may include orrepresent an entire of, or a part of, a musical piece. Further, in caseof a musical piece that includes symbols for multiple musicalinstruments, the respective sequence may represent or include a sequenceof symbols that is adapted only for the specific musical instrument,such as only for the piano 83. Each of, or part of, the musical pieces88 a-88N may comprise, may be part of, or may consist of, a song, avocal music work, or a classical music work, such as an Aria, anCadenza, an Concerto, an Movement, an Overture, an Opera, an Sonata, anChamber music, or an Symphony. Alternatively or in addition, each of, orpart of, the musical pieces 88 a-88N may comprise, may be part of, ormay consist of, a popular music work, such as a song, a dance, or aFunk, Country, Latin, Reggae, Hip-hop, or Polka music genre.

An example of the system 80 operation is presented as a flow chart 90shown in FIG. 9 . As part of an “Obtain User ID” step 91, the clientdevice 35 identifies the person, such as the user 36, that operate thedevice. For example, such identifier or username may be inserted by theuser 36 as part of a logging in to relevant application, for example byusing the input component 38. Preferably, such user identification isthe identical or directly and unambiguously corresponds to the one ofthe identifiers stored as ‘User ID’ column 86 a in the table 86. Theobtained user identifier is sent to the server device 23 a over theInternet 22 via the wireless antenna 29 and the wireless transceiver 28,as part of a “Send User ID” step 92, and received by the server device23 a as part of a “Receive User ID” step 92 a. Using the table 86, theserver device 23 a locates the received user identifier in the table 86,and determines the associated skill level value. For example, in casethe user identifier is “User #3”, the user 84 c is located, and a skilllevel value of ‘2’ is determined.

In one example, the musical piece to be played for the learning sessionis selected by the user 36, such as by using the input component 38, andsent to the server device 23 a with the user identifier as part of the“Send User ID” step 92, and received by the server device 23 a as partof the “Receive User ID” step 92 a. Alternatively or in addition, theserver device 23 a selects a musical piece to play, such as from thetable 89, preferably based on the user 36 determined skill level value.For example, is case of a User #3 84 c and a skill level value of ‘2’,the Piece #2 88 b may be selected since Allegro tempo is best suited forthis skill level. Alternatively or in addition, the last piece that wasselected by the user 36, or the last piece played by the user 36, isselected for further playing and practicing. Alternatively or inaddition, a piece (or part thereof) that was stored at a previouspractice session since it was not well played (such as identified andstored as part of a “Store Part ID” step 135 or 135 a described below),may be selected by the system, or recommended or offered to be selectedby the user 36. The determining of the respective skill level value ofthe user and the selecting of the musical piece to be played areperformed as part of a “Determine Skill Level & Prepare Musical Piece”step 93. In the case where the musical piece is selected or recommendedby the system, the system may use a comparison between the user skilllevel value 86 b and the complexity/difficulty level 87 e. For example,the user #1 84 a is associated with a skill level value of ‘1’, and thusonly the Piece #N 88N that is associated with the complexity/difficultylevel of ‘1’ may be suggested or selected, since all other pieces in thetable 89 are associated with higher complexity/difficulty level values.Similarly, the user #3 84 c is associated with a skill level value of‘2’, and thus only the Piece #N 88N that is associated with thecomplexity/difficulty level of ‘1’ and the Piece #1 88 a that isassociated with the complexity/difficulty level of ‘2’ may be suggestedor selected, since all other pieces in the table 89 are associated withhigher complexity/difficulty level values. In such a case, the Piece #188 a may be preferred by the system or the user since thecomplexity/difficulty level of ‘1’ associated with the Piece #N 88N maybe too easy, and not be challenging enough or promoting the progress ofthe student. Further, the user #4 84 d is associated with a skill levelvalue of ‘5’, and thus all the pieces in the table 89 are suitable, withpreference to the Piece #4 88 d that is associated with the same level‘5’.

Upon selecting a musical piece as part of the “Determine Skill Level &Prepare Musical Piece” step 93, the respective music sheet file, such asthe sheet2.pdf for the Piece #2 88 b, and the respective sequence ofmusical symbols is sent. In one example, the musical symbols of thesequence are sent one at a time. As part of an “End of Piece?” step 94,the server device 23 a checks whether all the symbols of the relatedsequence have been sent. As long as the sequence has not been sent infull, the next musical symbol in the sequence is sent as part of a “SendNext Symbol” step 95, and received by the client device 35 as part of a“Receive Next Symbol” step 95 a. The received symbol is displayed by thedisplay 81 as part of a “Display Next Symbol” step 96, such as thesymbol 85 b shown in FIG. 8 a . Alternatively or in addition to sendingthe musical symbols of the sequence one at a time, part of, or all of,the symbols of the sequence are sent together, displayed together on thedisplay 81, such as the sequence 85 a shown in FIG. 8 a , and thesymbols are marked one at a time, such as by the arrow 85 c shown inFIG. 8 a . In such a case, an identification of the next symbol in thesequence is performed as part of a “Identify Next Symbol” step 103, sentto the client device 35 as part of the “Send Next Symbol” step 95,received by the client device 35 as part of the “Receive Next Symbol”step 95 a, and the “Display Next Symbol” step 96 includes moving themarker (such as the arrow 85 c) to the received next symbol.

In response to notifying the user 36 of the next musical symbol to beplayed, the player 36 is expected to timely and correctly play the piano83 to produce the required sound that corresponds to the symbol that isdisplayed (or marked) on the display 81. As part of a “Capture Sound”step 97, the microphone 82 captures the received sound produced by thepiano 83, and after digitization and encapsulation the digitizedcaptured sound is sent, by the client device 35 as part of a “SendSound” step 98, to the server device 23 a over the Internet 22 via theantenna 29 and the wireless transceiver 28, where it is received as partof a “Receive Sound” step 98 a.

The received sound is analyzed as part of an “Analyze Sound” step 104.In one example, the received sound is obtained as a PCM audio data, andas part of a “Correct Sound?” step 99, the sound received as part of the“Receive Sound” step 98 a is compared to the sound expected as a resultof the musical symbol sent as part of the “Send Next Symbol” step 95 andidentified as part of the “Identify Next Symbol” step 103. In theexample of the piano 83 that includes 88 piano keys, the received soundstream is analyzed as part of the “Analyze Sound” step 104 to map thereceived sound to one (or more) played key out of the 88 piano keys, andas part of the “Correct Sound?” step 99, the mapped key (or multiplekeys) is checked versus the key (or multiple keys) that is representedby the displayed musical symbol displayed as part of the “Display NextSymbol” step 96, for example according to the keys/symbols mappingscheme shown in FIG. 6 a.

For example, the frequency response, the pitch, or the tonal or harmonicvalues of the received sound are compared to the expected values thatcorresponds to the sent symbol. In the case where it is determined aspart of the “Correct Sound?” step 99 that the right key (and/or pedal)was pressed, and assuming that it was determined that this last symbolwas not the last symbol of the sequence, as part of the “End of Piece?”step 94, the next symbol is repeatedly selected and sent as part of the“Send Next Symbol” step 95, after a delay as part of the “Delay” step106.

The analysis of the captured sound signal as part of the “Analyze Sound”step 104 may be based on feature extraction using time-domain, frequencydomain, or a combination thereof. The analysis is configured to handleharmonic instruments, such as the piano 83, where a single note or chordis composed of multiple frequencies (tones), playing multiple keys ornotes together simultaneously, in an environment of environment noise,such as air conditioner, operating television set, street noise, andother general background noise in a building environment.

The analysis may use, may be based on, or may include Mel-FrequencyAnalysis, such as calculating the Mel-Frequency Cepstral Coefficients(MFCC). Alternatively or in addition, the analysis may use, may be basedon, or may include Linear Predictive Coding (LPC), such as calculatingthe LPC coefficients. The extracted features may be, or may be based on(such as a function of), part of, or all of, the Mel-Frequency CepstralCoefficients (MFCC), the LPC coefficients, or any combination thereof.Further, the analysis may use signal dominant frequency bands or a phaseof the signal.

The feature extraction may be based on, or use, time-domain analysis,frequency-domain analysis, or both. A frequency-domain representationcan also include information on the phase shift that must be applied toeach sinusoid in order to be able to recombine the frequency componentsto recover the original time signal. An example for such conversion maybe the Fourier transform, which converts the time-function into a sum ofsine waves of different frequencies, each of which represents afrequency component. The ‘spectrum’ of frequency components is thefrequency domain representation of the signal. The inverse Fouriertransform converts the frequency domain function back to a timefunction. A spectrum analyzer is the tool commonly used to visualizereal-world signals in the frequency domain. Some specialized signalprocessing techniques use transforms that result in a jointtime-frequency domain, with the instantaneous frequency being a key linkbetween the time domain and the frequency domain.

There are a number of different mathematical transforms that may be usedto analyze time domain functions and are referred to as “frequencydomain” methods. The most common transforms are Fourier series, Fouriertransform, Laplace transform, Z transform, and Wavelet transform. TheFourier transform of a periodic signal only has energy at a basefrequency and its harmonics. Another way of saying this is that aperiodic signal can be analyzed using a discrete frequency domain.Dually, a discrete-time signal gives rise to a periodic frequencyspectrum. Combining these two, if we start with a time signal that isboth discrete and periodic, we get a frequency spectrum that is bothperiodic and discrete. This is the usual context for a discrete Fouriertransform. Converting to frequency domain may include, may use, or maybe based on, one or more of the methods described in articles by BoualemBoashash published in Proceedings of the IEEE, Vol. 80, No. 4, April1992 (0018-9219/92$03.00, 1992 IEEE) entitled: “Estimating andInterpreting The Instantaneous Frequency of a Signal—Part 1:Fundamentals”, and “Estimating and Interpreting The InstantaneousFrequency of a Signal—Part 2: Algorithms and Applications”, and in anarticle by Jonatan Lerga (of University of Rijeka) entitled: “Overviewof Signal Instantaneous Frequency Estimation Methods”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

The “Analyze Sound” step 104 may use, or may be based on, using afeature vector called the Enhanced Pitch Class Profile (EPCP) thatintroduced for automatic chord recognition from the raw audio in anarticle by Kyogu Lee Published 2006 by the center for Computer Researchin Music and Acoustics Department of Music, Stanford University,entitled: “Automatic Chord Recognition from Audio Using Enhanced PitchClass Profile”, which is incorporated in its entirety for all purposesas if fully set forth herein. The Harmonic Product Spectrum is firstobtained from the DFT of the input signal, and then an algorithm forcomputing a 12-dimensional pitch class profile is applied to it to givethe EPCP feature vector. The EPCP vector is correlated with thepre-defined templates for 24 major/minor triads, and the templateyielding maximum correlation is identified as the chord of the inputsignal. The experimental results show the EPCP yields less errors thanthe conventional PCP in frame-rate chord recognition.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, chord recognition systems that typically comprise anacoustic model that predicts chords for each audio frame, and a temporalmodel that casts these predictions into labelled chord segments, asdescribed in an article by Filip Korzeniowski and Gerhard Widmerpresented in the 19th International Society for Music InformationRetrieval Conference, Paris, France, 2018, by the Institute ofComputational Perception, Johannes Kepler University, Linz, Austriaentitled: “IMPROVED CHORD RECOGNITION BY COMBINING DURATION AND HARMONICLANGUAGE MODELS”, which is incorporated in its entirety for all purposesas if fully set forth herein. Temporal models have been shown to onlysmooth predictions, without being able to incorporate musicalinformation about chord progressions. Recent research discovered that itmight be the low hierarchical level such models have been applied to(directly on audio frames) which prevents learning musicalrelationships, even for expressive models such as Recurrent NeuralNetworks (RNNs). However, if applied on the level of chord sequences,RNNs indeed can become powerful chord predictors. In this paper, wedisentangle temporal models into a harmonic language model—to be appliedon chord sequences—and a chord duration model that connects thechord-level predictions of the language model to the frame-levelpredictions of the acoustic model. In our experiments, we explore theimpact of each model on the chord recognition score, and show that usingharmonic language and duration models improves the results.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, a system and method for improving musical educationthrough use of a game is disclosed in U.S. Pat. No. 9,492,756 toIzkovsky et al. entitled: “System and method for analyzing a digitalizedmusical performance”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The method includes the steps of:receiving electrical signals associated with a musical piece provided bya user of the game; converting the electrical signals into digitalsamples; and analyzing the digital samples with use of auxiliaryinformation, for purposes of improving signal analysis accuracy,resulting in determining various parameters of the musical pieceprovided by the user, wherein the auxiliary information is a-priori datarelated to at least one element selected from the group consisting of amusical instrument being played, a technical environment, the game, andinformation regarding the user of the game.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, extracting Chroma features, such as described in anarticle entitled: “Analyzing Chroma Feature Types for Automated ChordRecognition” by Nanzhu Jiang, Peter Grosche, Verena Konz, and MeinardMuller, published 2011 Jul. 22-24 in the AES 42ND INTERNATIONALCONFERENCE, Ilmenau, Germany, which is incorporated in its entirety forall purposes as if fully set forth herein. The computer-based harmonicanalysis of music recordings with the goal to automatically extractchord labels directly from the given audio data constitutes a major taskin music information retrieval. In most automated chord recognitionprocedures, the given music recording is first converted into a sequenceof chroma-based audio features and then pattern matching techniques areapplied to map the chroma features to chord labels. In this paper, weanalyze the role of the feature extraction step within the recognitionpipeline of various chord recognition procedures based on templatematching strategies and hidden Markov models. In particular, we reporton numerous experiments which show how the various procedures depend onthe type of the underlying chroma feature as well as on parameters thatcontrol temporal and spectral aspects.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, determining various parameters of the musical piecebeing played by the user. Examples of such parameters include, but arenot limited to, the tone being played, the intensity of the tone, alsoreferred to as the volume, the duration of the tone, and any additionalcharacteristics that would assist in analyzing the playing of themusical piece by the user. Such additional characteristics may include,for example, instrument-specific measures, such as the volume of airthat the user puts into a woodwind instrument, the accuracy of thetiming and tempo a user plays a percussion instrument, the intensitythat the user presses a piano key, or plays a guitar string, or othercharacteristics.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, estimating the fundamental frequencies of concurrentmusical sounds, as described in an article by A. P. Klapuri entitled:“Multiple fundamental frequency estimation based on harmonicity andspectral smoothness” published in IEEE TRANSACTIONS ON SPEECH AND AUDIOPROCESSING, VOL. 11, NO. 6, NOVEMBER 2003 [11(6), 804-816, 2003], whichis incorporated in its entirety for all purposes as if fully set forthherein. The method is based on an iterative approach, where thefundamental frequency of the most prominent sound is estimated, thesound is subtracted from the mixture, and the process is repeated forthe residual signal. For the estimation stage, an algorithm is proposedwhich utilizes the frequency relationships of simultaneous spectralcomponents, without assuming ideal harmonicity. For the subtractionstage, the spectral smoothness principle is proposed as an efficient newmechanism in estimating the spectral envelopes of detected sounds. Withthese techniques, multiple fundamental frequency estimation can beperformed quite accurately in a single time frame, without the use oflong-term temporal features. The experimental data comprised recordedsamples of 30 musical instruments from four different sources. Multiplefundamental frequency estimation was performed for random sound sourceand pitch combinations. Error rates for mixtures ranging from one to sixsimultaneous sounds were 1.8%, 3.9%, 6.3%, 9.9%, 14%, and 18%,respectively. In musical interval and chord identification tasks, thealgorithm outperformed the average of ten trained musicians. The methodworks robustly in noise, and is able to handle sounds that exhibitinharmonicities. The inharmonicity factor and spectral envelope of eachsound is estimated along with the fundamental frequency.

Alternatively or in addition, the “Analyze Sound” step 104 may use, ormay be based on, a conceptually simple and computationally efficientfundamental frequency (F0) estimator for polyphonic music signals, asdescribed in an article by A. P. Klapuri entitled: “Multiple FundamentalFrequency Estimation by Summing Harmonic Amplitudes” published 2006 bythe Institute of Signal Processing, Tampere University of Technology,University of Victoria, which is incorporated in its entirety for allpurposes as if fully set forth herein. The studied class of estimatorscalculate the salience, or strength, of a F0 candidate as a weighted sumof the amplitudes of its harmonic partials. A mapping from the Fourierspectrum to a “F0 salience spectrum” is found by optimization usinggenerated training material. Based on the resulting function, threedifferent estimators are proposed: a “direct” method, an iterativeestimation and cancellation method, and a method that estimates multipleFOs jointly. The latter two performed as well as a considerably morecomplex reference method. The number of concurrent sounds is estimatedalong with their FOs.

The “Delay” step 106 is operative to set the pace or tempo to therequired one. Assuming the required pact or tempo is Allegro,corresponding to 120 bpm (a beat every 500 milliseconds (ms)), such asfor the Piece #2 88 b as part of table 89 shown in FIG. 8 b , then aspart of the “Delay” step 106, a delay is induced to ensure that the paceor rate of displaying musical symbol to be played is 500 ms. Forexample, in case there is a delay of 100 milliseconds from the time ofthe “Send Next Symbol” step 95 (or from the “Identify Next Symbol” step103) to the end of checking of the correctness of the sound as part ofthe “Correct Sound?” step 99, a delay of an additional 400 ms is inducedto arrive at the total of 500 ms.

In one example, the pace of displaying musical symbols on the display 81to be played by the user 36 is reduced according to the skill levelvalue, to better fit to the capability of the player 36, and to avoidfrustrations due to hard to comply with speedy pace or tempo. In theexample of skill level 1 to 5, where ‘1’ is a beginner or novice and ‘5’is an expert level, the pace may be lowered accordingly. For example, incase of a tempo of Allegro that corresponds to 120 bpm, the Delay aspart of the “Delay” step 106 is adapted so that in response to a skilllevel of ‘1’ the affective bpm is reduced to 40 bpm (a total delay of1.5 second), in response to a skill level of ‘2’ the affective bpm isreduced to 60 bpm (a total delay of 1 second), in response to a skilllevel of ‘3’ the affective bpm is reduced to 80 bpm (a total delay of750 milliseconds), in response to a skill level of ‘4’ the affective bpmis reduced to 100 bpm (a total delay of 600 milliseconds), and inresponse to a skill level of ‘5’ the nominal tempo of 120 bpm isretained (a total delay of 500 milliseconds). In another example, inresponse to a skill level of ‘1’ the affective bpm is reduced to 20% ofthe nominal value of 120 bpm−24 bpm (a total delay of 2.5 second), inresponse to a skill level of ‘2’ the affective bpm is reduced to 40% ofthe nominal value of 120 bpm-48 bpm (a total delay of 1.25 seconds), inresponse to a skill level of ‘3’ the affective bpm is reduced to 60% ofthe nominal value of 120 bpm−72 bpm (a total delay of 833 milliseconds),in response to a skill level of ‘4’ the affective bpm is reduced to 80%of the nominal value of 120 bpm−96 bpm (a total delay of 625milliseconds), and in response to a skill level of ‘5’ the affective bpmis not changed and is 100% of the nominal value of 120 bpm.

Alternatively or in addition to the linear functions exampled herein,the correspondence between the skill level value and the displayed paceor tempo may be a non-linear function, such as where in response to askill level of ‘1’ the affective bpm is reduced to 40% of the nominalvalue, in response to a skill level of ‘2’ the affective bpm is reducedto 65% of the nominal value, in response to a skill level of ‘3’ theaffective bpm is reduced to 75% of the nominal value, in response to askill level of ‘4’ the affective bpm is reduced to 90% of the nominalvalue, and in response to a skill level of ‘5’ the affective bpm is notchanged and is 100% of the nominal value.

Upon completion of the playing of the entire sequence, such asfinalizing the playing of the entire musical piece, as checked byarriving at the last symbol of the sequence in the “End of Piece?” step94, the sending of symbols stops and finalizing actions, such asupdating the skill level value 86 b of the specific user 86 a in thetable 86 as part of a “Update Skill Level” step 102, which may be basedon the number of errors that are detected as part of the “CorrectSound?” step 99. In one example, the practicing session is ended as partof an “End” step 105. For example, a low threshold for the numbers oferror in the played piece may be pre-defined, such as a different valuefor each skill level value. In case the number of errors is zero or lessor less than the threshold value, the skill level value may be raised,such as from ‘3’ to ‘4’. Similarly, a high threshold for the numbers oferror in the played piece may be pre-defined, such as a different valuefor each skill level value. In case the number of errors is more thanthe threshold value, the skill level value may be lowered, such as from‘3’ to ‘2’. Similarly, in case where the number of errors is between thelow and high thresholds, the skill level value remains unchanged.

A threshold, such as the low or high threshold, may be based on absolutenumber of errors, such as above 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50errors, or below 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50 errors.Alternatively or in addition, a threshold, such as the low or highthreshold, may be based on the relative errors versus the total numberof symbols, such as being at least 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%,5%, 8%, 10%, 15%, 20%, 30%, or 50% errors of the total number of symbolsin the sequence, or being less than 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%,5%, 8%, 10%, 15%, 20%, 30%, or 50% errors of the total number of symbolsin the sequence.

While in the flow chart 90 the “Delay” step 106 is executed by theserver device 23 a, this functionality may equally be performed at theclient device 35, as exampled in a flow chart 100 shown in FIG. 10 .Further, this function may be divided or cooperated between bothdevices.

In one example, the sequence of symbols that is displayed and checked inthe flow chart 90 in FIG. 9 describes a practice that involves playingof the entirety of a musical piece. However, as an alternative or inaddition, the sequence of symbols may equally represent only part ofmusical piece. In such a case, the items in rows 88 in the table 89 mayrefer to part of a musical piece, and not the entire piece.

A part of, or an entire of, musical piece may represent only a fractionof the piece length that is a partial duration of the entire piece. Forexample, in case of a musical piece that requires 20 minutes of playing,a part of it may be associated with 25% of the piece, specifically 5minutes of playing. The 5-minutes duration part may be located anywherein the piece, such as in the beginning of the piece (0-5 minutes), atthe end of the piece (15-20 minutes), or in the middle of the piece,such as 5-10, or 10-15 minutes, parts. Similarly, the segment may beanywhere during the musical piece, such as in a 2-7, 8-13, 9.2-14.2minutes of the piece playing. Similarly, the part duration may bedefined in a relative term, such as at least 0.1%, 0.2%, 0.3%, 0.5%,0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%,70%, 80%, or 90%, of the playing time of the musical piece, or may beless than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%,25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the musical pieceplaying duration. Further, a part of a musical piece may be defined byits absolute duration that is less than the entire musical piece playingduration, such as at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40,50, 60, 80, 100, 150, 200, 300, 500, or 1000 seconds, or may be lessthan 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200,300, 500, 1000 or 2000 seconds.

As an alternative or in addition, the duration of the part of themusical piece may be measured by means of number of symbols. Forexample, a musical piece may include 1200 symbols. In one example, apart may include a sequence of 100 symbols that may be the first 100symbols (1-100), the last 100 symbols (1101-1200), or any sub-sequencein the middle of the piece, such as the symbols at the locations of301-400, 979-1078, 1124-1223. Similarly, the part duration may bedefined in a relative term, such as at least 0.1%, 0.2%, 0.3%, 0.5%,0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%,70%, 80%, or 90%, of the number of symbols for the full playing time ofthe musical piece, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%,2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%,or 95% of the number of symbols relating in the entire musical pieceplaying duration. Further, a part of a musical piece may be defined byits absolute number of symbols (that is less than the entire musicalpiece playing duration), such as at least 1, 2, 3, 5, 8, 10, 12, 15, 20,25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000 symbols, or maybe less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100,150, 200, 300, 500, 1000 or 2000 symbols.

In one example, the duration of a part, wither by means of time durationor by means of number of symbols, may be determined in response to theskill level value of a specific user 84. A shorter part may be moresuitable to lower skill level values compared with a longer parts thatmay be more suitable to higher skill level values, in order to reducefrustration of the player. For example, a user such as the User #1 84 athat is associated with skill level value of ‘1’, may be assigned forpractice with a part that is shorter than a user, such as the User #2 84b that is associated with skill level value of ‘3’, as described in thetable 89.

As described in the flow chart 90 shown in FIG. 9 , the skill levelvalue 86 b of the specific user 86 a in the table 86 is updated as partof a “Update Skill Level” step 102, at the end of playing a piece asdetermined in the “End of Piece?” step 94. In this example, the updatedskill level value influences the playing pace only on a next practicingsession where a new musical piece (or the same one) is selected for useas part of the “Determine Skill Level & Prepare Musical Piece” step 93.Such scenario may frustrate the user if there is a substantial mismatchbetween the user actual skill level 86 b and the skill level stored inthe user database 31 a.

Alternatively or in addition, the modification of the skill level value86 b and the associated playing pace may be performed in specified timeswithin the playing period, without disrupting the practicing flow oraffecting the progress momentum. For example, a musical piece may bepartitioned into multiple parts that are sequentially played, and theassessment of the user 36 skill level value, and the associated playingpace, are updated at the end of each part, in a fluent real time manner,where the transition is not felt by the player 36. Such mechanismreduces the user 36 frustration, and allows for quick, on-the-fly,real-time adaptation of the user 36 skill level 86 b and the respectiveplaying pace along with a continuous practicing session.

A part-based updating of the skill level value 86 b (and the associatedpace) is described in a flow chart 110 shown in FIG. 11 . Each of themusical pieces 88 in the table 89 is partitioned into multiple parts.After selecting a musical piece as part of the “Determine Skill Level &Prepare Musical Piece” step 93, a part of the selected musical piece isselected as part of a “Select Next Part” step 111. For example, the partthat begins the piece may be first selected, followed sequentially bythe other parts. As long as the all symbols of the selected part are notyet played, as determined in a “End of Part ?” step 112, the playing ofthe selected part continues, similar to the process described in theflow chart 90. After displaying (and playing) of the last symbol of theselected part, as detected in the “End of Part ?” step 112, it isdetermined, similar to the flow chart 90, whether the entire pieceplaying is completed, for example if the part that is ending of thepiece has been played, as part of the “End of Piece ?” step 94. In casethe musical piece has not been entirely played, the user skill levelvalue is updated as part of the “Update Skill Level” step 102, based onthe number of errors detected as part of the “Correct Sound ?” step 99.The next part to be played is then selected as part of the “Select NextPart” step 111, and the next selected part is played in a pace thatcorresponds to the user's updated skill level value. In such a scheme, aproper error-free (or minimally errored) played part may result inincreasing of the user skill level value and accordingly increasing ofthe pace of playing of the parts, and an excessive error parts mayresult in lowering of the user skill level value and accordinglyreducing the pace of playing of the parts.

A musical piece may be partitioned into any number of parts, such as atleast 2, 3, 4, 5, 7, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, or 100parts, or may be partitioned into less than 3, 4, 5, 7, 10, 12, 15, 20,25, 30, 40, 50, 60, 80, 100, or 200 parts. The partition may besequential in the musical piece, where each part (except the first partat the beginning of the piece) is following another part in the symbolsequence, or may be non-sequential, for example where there are symbolsthat do not belong to any one of the parts. Further, the partition maybe non-overlapping, wherein each specified symbol in the piece isincluded in no more than a single part, or may be overlapping, whereinat least one specific symbol is included in two or more parts. In oneexample, the parts are of different playing duration or includedifferent number of symbols. In another example, at least two of theparts, or all of the parts, are of the same playing duration or includethe same number of symbols.

The partition may be time based, wherein one of the parts, at least twoof the parts, or each of the parts, may be at least 0.1%, 0.2%, 0.3%,0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%,50%, 60%, 70%, 80%, or 90%, of the playing time of the musical piece, ormay be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%,20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the musicalpiece playing duration. Similarly, one of the parts, at least two of theparts, or each of the parts, may be of a duration of at least 1, 2, 3,5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500,or 1000 seconds, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30,40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 seconds.

As an alternative or in addition, the partition may be symbol based,wherein one of the parts, at least two of the parts, or each of theparts, may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%,5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, ofthe number of symbols for the full playing time of the musical piece, ormay be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%,20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the numberof symbols relating in the entire musical piece playing duration.

While exampled in the flow chart 110 in FIG. 11 that the user 36 isvisually instructed how to play the musical instrument (such as thepiano 83) using the display 81 in the client device 35, audibleinstruction or cues may be equally used, as an alternative or inaddition to visual interaction with the user 36, as exampled in a flowchart 110 a shown in FIG. 11 a . As part of an “Identify Instruction”step 113, an audible instruction or cue is identified. In one example,the audible instruction may include, or may be associated with, anaudible identifier of the symbol (such as a note or chord) that isidentified as part of the “Identify Next Symbol” step 103, such as “playnote X” or “chord Y”. Alternatively or in addition, the audibleinstruction may include, or may be associated with, other instructionsor cues pertaining to how to play the musical instrument 83. Forexample, each key of the piano 83 may be numbered, and the voice commandmay include identifying of the key number, such as ‘key five’ or ‘playthree and seven together’. The identified cue or instruction may then besent over a communication path 95 c, that may be the same as, identicalto, similar to, or part of, the communication path 95 b, and delayed aspart of a “Delay” step 106 a, which may be the same as, identical to,similar to, or part of, the “Delay” step 106, and then vocalized to besounded to the user 36, such as by the sounder 78, as part of a“Vocalize Instruction” step 114. In case where instruction for playingthe musical instrument 83 are provided by both the audible sounding aspart of the “Vocalize Instruction” step 114 and the “Display NextSymbol” step 114, both instructions or cues should be synchronized andcoordinated, in order not to confuse the user 36. The “Delay” step 106 amay be used to provide a delay that provide coordination between bothinstructions (or cues) provided to the user 36. The audio instruction isshown in the flow chart 110 a in FIG. 11 a to be determined in theserver device 23 a. Alternatively or in addition, the determining aspart of the “Identify Instruction” step 113 may be performed in theclient device 35, in response to the received next symbol in thecommunication path 95 b.

The sounding of the instructions or ques as part of the “VocalizeInstruction” step 114 may use, may be based on, or may comprise, and thesounded human speech may be produced using a hardware, software, or anycombination thereof, of a speech synthesizer, which may beText-To-Speech (TTS) based. The speech synthesizer may be aconcatenative type, using unit selection, diphone synthesis, ordomain-specific synthesis. Alternatively or in addition, the speechsynthesizer may be a formant type, and may be based on articulatorysynthesis or Hidden Markov Models (HMM) based. Further, any speechsynthesizer herein may be based on, or may use, any of the schemes,techniques, technologies, or arrangements described in the bookentitled: “Development in Speech Synthesis”, by Mark Tatham andKatherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN:0-470-85538-X, in the book entitled: “Speech Synthesis and Recognition”by John Holmes and Wendy Holmes, 2^(nd) Edition, published 2001 ISBN:0-7484-0856-8, in the book entitled: “Techniques and Challenges inSpeech Synthesis—Final Report” by David Ferris [ELEC4840B] publishedApr. 11, 2016, or in the book entitled: “Text-to-Speech Synthesis” byPaul Taylor [ISBN 978-0-521-89927-7] published 2009 by CambridgeUniversity Press, which are all incorporated in their entirety for allpurposes as if fully set forth herein.

In one example, the interaction with the user 36 is enhanced byproviding feedback that is based on the actual analysis of a practicesession, such as playing a part of, or an entire of, a musical piece, tothe user 36. For example, in case of no errors, or tolerable number oferrors (that may correspond to the user skill level) in a practicesession, a positive feedback may be provided or notified to the uservisually or audibly, such as ‘Good work’, ‘Great’, ‘Terrific’,‘Impressive’, ‘Way to go’, ‘Exactly right!’ or ‘No errors’, and in caseof upgrading the skill level value as part of “Update Skill Level” step102, accordingly notifying the user 36 in a message such as ‘Skill levelupgraded’ or ‘Congratulations, you got it right!’. Similarly, in case ofpoor performance, such as too many errors in playing, a negative orencouraging feedback may be provided or notified to the user visually oraudibly, such as ‘Keep practicing’, ‘Too many errors’, ‘Don't give up’,or ‘Come on! You can do it!’. A feedback mechanism is exampled in theflow chart 110 a shown in FIG. 11 a.

Upon completing the practice of a part of a musical piece as determinedas part of the “End of Part” step 112, in addition to updating the skilllevel value as part of the “Update Skill Level” step 102, a feedbackmessage is determined as part of an “Identify Feedback” step 115. Themessage identified to be provided or notified to the user 36 may bebased on the number of errors detected through the practice session,such as part of the “Error Action” step 101. For example, the messagemay be base on any score, such as the cumulative number of errors Ndetermined in the “N←N+1” step 132 described herein, may be based oncomparing the number of cumulative errors to a threshold, such as in the“N>Nthres ?” step 133 described herein, may be based on the decision tochange the skill level value as part of the “Update Skill Level” step102, or any combination thereof. In one example, the message to benotified or otherwise provided to the user 36 is determined as part ofthe “Identify Feedback” step 115 in the client device 35. Alternativelyor in addition, the message to be notified or otherwise provided to theuser 36 may be determined as part of the “Identify Feedback” step 115 inthe server device 23 a, and sent to the client device 35 over acommunication path 95 d, and then the received message is then displayedon the display 81 as part of the “Display Feedback” step 96 b.

As an alternative to, or in addition to, visually displaying thefeedback message on the display 81, the feedback message may be soundedby the sounder 78 as part of a “Vocalize Feedback” step 114 a. Thesounding of the feedback message as part of the “Vocalize Feedback” step114 a may use, may be based on, or may comprise, and the sounded humanspeech may be produced using a hardware, software, or any combinationthereof, of a speech synthesizer, which may be Text-To-Speech (TTS)based. The speech synthesizer may be a concatenative type, using unitselection, diphone synthesis, or domain-specific synthesis.Alternatively or in addition, the speech synthesizer may be a formanttype, and may be based on articulatory synthesis or Hidden Markov Models(HMM) based. Further, any speech synthesizer herein may be based on, ormay use, any of the schemes, techniques, technologies, or arrangementsdescribed in the book entitled: “Development in Speech Synthesis”, byMark Tatham and Katherine Morton, published 2005 by John Wiley & SonsLtd., ISBN: 0-470-85538-X, in the book entitled: “Speech Synthesis andRecognition” by John Holmes and Wendy Holmes, 2^(nd) Edition, published2001 ISBN: 0-7484-0856-8, in the book entitled: “Techniques andChallenges in Speech Synthesis—Final Report” by David Ferris [ELEC4840B]published Apr. 11, 2016, or in the book entitled: “Text-to-SpeechSynthesis” by Paul Taylor [ISBN 978-0-521-89927-7] published 2009 byCambridge University Press, which are all incorporated in their entiretyfor all purposes as if fully set forth herein.

While exampled regarding notifying a feedback message to the user 36 asthe end of a part of musical piece, such notifying may be activated,alternatively or in addition, at the end of the entire musical piece,shown by a dashed line in the flow chart 110 a, where upon determiningthat the entire piece was played as part of the “End of Piece?” step 94,the feedback message is identified as part of the “Identify Feedback”step 115, and notified to the user 36 as part of the “Display Feedback”step 96 b or the “Vocalize Feedback” step 114 a.

In the example shown in the arrangement 80 a shown in FIG. 8 a , thesymbols to be played are visually displayed to the user 36 using adisplay 81 in the client device 35. Alternatively or in addition, thedisplay 81 may be a projector. Such projector may indicate byilluminating to the user 36 how to play the musical instruments. In theexample of the piano 83, the projector may illuminate on the pianokeyboard, identifying to the user 36 by illuminating what key (ormultiple keys) are to be next played. The projector serving as thedisplay 81 may consist of, or may comprise, an Eidophor projector,Liquid Crystal on Silicon (LCoS or LCOS) projector, an LCD projector, anMEMS projector, a virtual retinal display, or a Digital Light Processing(DLP™) projector. Further, the display 81 or the display screen mayconsist of, or may comprise, a video display, such as a 3D videodisplay, that supports Standard-Definition (SD) or High-Definition (HD)standards, and may be capable of scrolling, static, bold or flashing thepresented information.

Upon detecting an error as part of the “Correct Sound?” step 99, variouserror related actions may be performed, individually or in combination,as part of the “Error Action” step 101. The various alternatives forprogress of playing practice session in response to an error aredescribed in a flowchart 120 shown in FIG. 12 . In one example, shown asdashed line 121, there is no stopping or delaying of the progress of thesession, and even upon detecting of an error in playing a displayedmusical symbol, the progress relating to the selecting and displaying ofthe next symbol is not affected, delayed, or changed in any way, andcontinues as part of the “End of Part” step 112. Such mechanism may beuseful in particular for a low skill level player, who is expected tomake a lot of errors, and any stop of the flow may create frustrationand may result in negative user experience. As an alternative, adetection of an error (or in case of multiple errors) may lead to endingof the practice session as part of the “END” step 105, as shown by adashed line 122. In case of multiple errors detected (or even in case ofa single error), rather than ending the practice session altogether, theresponse may be to move to the next part as part of the “Select NextPart” step 111, as shown by a dashed line 124. In order to better adaptto the player 36 skill level, the associated skill level value 86 b maybe updated as part of the “Update Skill level” step 102, and only thenthe system may progress the next part as part of the “Select Next Part”step 111, where the pace associated with the updated skill level valueis used. The selection between the four alternatives progress paths 121,122, 123, and 124 may be based on pre-configuration logic that may bebased on the user 36 skill level value and the level or the playing,such as based on a cumulative number or errors rather than based on asingle error.

Alternatively or in addition, the selection between the fouralternatives progress paths 121, 122, 123, and 124 may be based on theuser 36 setting, as part of a configuration, installation, or loggingprocedure, for example by using the input component 38. Alternatively orin addition, the selection between the four alternatives progress paths121, 122, 123, and 124 may be based on the user 36 setting upon theoccurrence of an error (or of a series of errors), and the system mayrefer to the player 36 for a guidance how to progress. In such aconfiguration, which progress is shown by a dashed line 127, the serverdevice 23 a send an error message to the client device 35 as part of a“Send Error” step 128, and the error message is received by the clientdevice 35 as part of a “receive Error” step 128 a, and then the errormessage and the optional progress paths, such as to continue as beforeaccording to the path 121, to stop the session according to the path122, to continue to the next part according to the path 124, or toupdate the skill level value and then to continue to the next partaccording to the path 123, are displayed to the user 36 as part of a“Display Error” step 96 a. The player 36 may then select, such as byusing the input component 38, the preferred path to progress, and theselection is sent by the client device 35 to the server device 23 a aspart of a “Send Input” step 126, to be received by the server device 23a as part of a “Receive Input” step 126 a. The user 36 selection is thenused, as shown by a dashed line 127 a, as part of the “Error Action”step 101 for executing the progress path that was selected by the player36. In one example, as an alternative or in addition to visuallynotifying the user 36, the “Display Error” step 96 a may comprise anaudible notification to the user 36 using the speaker 78.

In one example, an action as part of the “Error Action” step 101 may beinitiated upon detecting a single error as part of the “Correct Sound ?”step 99. However, it is expected, in particular where low skill leveluser 36 is practicing, that many errors may occur. Hence, it may bepreferred to define an ‘error score’ that is based on the occurrence ofmore than a single error, and make use of cumulative counting of errorsin an entire musical piece, or any part thereof. Such an arrangement forcounting the number of errors in an entire musical piece is described ina flow chart 130 shown in FIG. 13 , where a counter ‘N’ is used to countthe number of errors. The N is reset to zero as part of a “N←O” step131, at the beginning of playing of the selected musical piece. Eachtime an error is detected as part of the “Correct Sound ?” step 99, theN counter is incremented as part of a “N←N+1” step 132. The cumulativenumber of errors N, the ‘error score’, may be used to determine how toupdate the skill level value in the “Update Skill Level” step 102, aftercompleting a part of the selected musical piece, as described herein.Upon completion of the playing of the musical piece, as determined aspart of the “End of Piece?” step 94, the errors score N is comparedversus a predefined threshold value Nthres, representing a maximumallowed number of errors that do not require a specific error actionresponse, as part of a “N>Nthres ?” step 133. In case the errors score Nis less than the set threshold, suggesting that the errors level istolerable, such as zero errors or under the defined minimum, and may notrequire a specific action as part of the “Error Action” step 101, andthus the piece playing session is ended in the “END” step 105.

In a case where it is determined that the number of errors (‘errorscore’) is above the defined threshold Nthres as part of the “N>Nthres?” step 133, various error relation actions may be performed. In oneexample, the identification, such as the row 88 of the musical piece isstored as part of a “Store Piece ID” step 135, for future use. Such usemay be recommending to practice again on this musical piece until therequired performance level is obtained, such as having an error score inthis piece that is lower than the Nthres specified threshold.

Further, the user 36 may be notified of the error score being above thethreshold, by sending an error message from the server device 23 a tothe client device 35 as part of the “Send Error” step 128, and receivingthe message by the client device 35 as part of the “Receive Error” step128 a. The message may then be notified to the user 36 as part of a“Notify User” step 134, which may use any output component 34 in theclient device 35. In one example, the notification to the user 36 aspart of the “Notify User” step 134 uses the display 81, visualizing theerror message to the user 36. For example, symbols that were notdetermined to be correct may be visualized as marked to the user 36,such as by highlighting these symbols, changing their color, flashingthe symbols, and similar markings. In one example, as an alternative orin addition to visually notifying the user 36, the “Notify User” step134 may comprise an audible notification to the user 36 using thespeaker 78.

While the flow chart 130 in FIG. 13 exampled the case of accumulatingerrors and acting accordingly in an entire musical piece, the errors mayequally be accumulated and acted upon in a part of a musical piece, asdescribed in a flow chart 130 a shown in FIG. 13 a . Similar to the flowchart 130, the N is reset to zero as part of a “N←0” step 131, at thebeginning of playing of the selected part of the selected musical piece.Each time an error is detected as part of the “Correct Sound ?” step 99,the N counter is incremented as part of a “N←N+1” step 132. Thecumulative number of errors N, the ‘error score’, may be used todetermine how to update the skill level value in the “Update SkillLevel” step 102, after completing the part of the selected musicalpiece, as described herein. Upon completion of the playing of the partof the musical piece, as determined as part of the “End of Part?” step112, the errors score N is compared versus a predefined threshold valueNthres, representing a maximum allowed number of errors that do notrequire a specific error action response, as part of a “N>Nthres ?” step133. In case the errors score N is less than the set threshold,suggesting that the errors level is tolerable, such as zero errors orunder the defined minimum, and may not require a specific action as partof the “Error Action” step 101, and thus the next part of the piece isplayed.

In a case where it is determined that the number of errors (‘errorscore’) is above the defined threshold Nthres as part of the “N>Nthres?” step 133, various error relation actions may be performed. In oneexample, the identification, such as the corresponding row 88 of themusical piece is stored as part of a “Store Part ID” step 135 a, forfuture use. Such use may be recommending to practice again on this partof the musical piece until the required performance level is obtained,such as having an error score in this part that is lower than the Nthresspecified threshold.

The predefined threshold value Nthres may be an absolute number thatconsists of a number of errored symbols, such as at least 1, 2, 3, 5, 8,10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000errored symbols, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30,40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 errored symbols.Alternatively or in addition, the predefined threshold value Nthres maybe a value relative to the total number of symbols in the musical pieceor the part thereof, such as at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%,1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%,or 90%, of the number of symbols for the entire musical piece (or thepart thereof), or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%,3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or95% of the number of symbols for the entire musical piece (or the partthereof).

In one example, inducing delay as part of the “Delay” step 106 may notbe required, as the musical piece arrangement, such as the associatedtempo, notes, or other decoration, already includes the required delay,thus no delay (affectively 0 delay) is required. No delay arrangement130 b is exampled in FIG. 13 b.

The server device 23 a is exampled herein as a dedicated device.Alternatively ort in addition, the functionalities provided by theserver device 23 a may be part of a cloud-based service, such as by acloud 158 shown in the arrangement 150 c in FIG. 15 c , where the serverdevice 23 a is implemented as Infrastructure as a Service (IaaS) or as aSoftware as a Service (SaaS) (or as a Platform as a Service (PaaS)),providing the benefits of a cloud based computing, such asconsolidation, virtualization, better cost, reliability, level ofservice, and automation. One benefit involves balancing betweenoff-premises and on-premises cloud storage options, or a mixture of thetwo options, depending on relevant decision criteria that iscomplementary to initial direct cost savings potential. The cloud 158may be a public cloud provider, such as by Amazon AWS, Microsoft Azureand Google GCP.

As shown in the flow chart 130 a in FIG. 13 a , the errors areaccumulated and counted as part of the “N←N+1” step 132, and the actionto be taken as part of Error Action Step 101 is based on the cumulativenumber of errors determined as part of the “N>Nthres ?” step 133. In oneexample, errors detected as part of the “Correct Sound ?” step 99 arefurther analyzed (such as in the “Analyze Sound” step 104) andcategorized into multiple groups, and the errors are accumulatedseparately in each group Ni, where (i) is the identifier of the group oferrors, as part of a “Ni←Ni+1” step 132 a shown as part of the flowchart 130 b. In such a case, all Ni counters are reset as part of a“Ni←0” step 131 a, and the determination regarding the action to performupon detecting too many errors may be defined as a function of thecumulative number of errors in all the error type groups, as part of a“f(Ni)>Nthres ?” step 133 a, as shown in FIG. 13 b . The function f(Ni)may be a linear function, such as weighted average function, where theassigned coefficients provides different weights for each errors group,or may be a non-linear function. The function f(Ni) may be based on, mayuse, or may comprise, a discrete, continuous, monotonic, non-monotonic,elementary, algebraic, linear, polynomial, quadratic, Cubic, Nth-rootbased, exponential, transcendental, quintic, quartic, logarithmic,hyperbolic, or trigonometric function.

The different error groups may be based on various erroring types ofcharacteristics, which may be based on difficulty or skillcharacteristics. For example, a group may include errors related torhythmic difficulty, relating to how irregularly musical notes andchords are located in time, and what is the “speed” of music (the rateat which notes/chords should be played), motoric difficulty, relating tohow difficult it is physically to produce the musical sounds using amusical instrument (usually mainly related to the complexity of thephysical arrangement of user's fingers), harmonic difficulty, relatingto how complex combinations of musical notes need to be playedsimultaneously (as chords) or sequentially (as very unpredictablemelodies), or expressivity difficulty, relating to the user need tocontrol and vary the loudness (“dynamics”) and timbre (“tone color”) ofthe musical sounds being produced.

Alternatively or in addition, the different error groups may be basedon, or may use, techniques, classifications, or features, described inan article entitled: “Statistical piano reduction controllingperformance difficulty” by Eita Nakamura and Kazuyoshi Yoshii[doi:10.1017/ATSIP.2018.18], published 2018 in SIP (2018), vol. 7, e13,in an article entitled: “Automatic System for the Arrangement of PianoReductions” by Shih-Chuan Chiu, Man-Kwan Shan, and Jiun-Long Huang,published 2009 in the 11th IEEE International Symposium on Multimedia,in an article entitled: “A Study on Difficulty Level Recognition ofPiano Sheet Music” by Shih-Chuan Chiu and Min-Syan Chen, publishedDecember 2012 in the ISM '12: Proceedings of the 2012 IEEE InternationalSymposium on Multimedia, in an article entitled: “SCORE ANALYZER:AUTOMATICALLY DETERMINING SCORES DIFFICULTY LEVEL FOR INSTRUMENTALE-LEARNING” by Véronique Sébastien, Henri Ralambondrainy, OlivierSébastien, and Noel Conruyt of IREMIA—Laboratoire d'Informatique et deMathématiques, EA2525 University of Reunion Island, Saint-Denis, Reunion(FRANCE), published October 2012 in 13th International Society for MusicInformation Retrieval Conference (ISMIR 2012), in a paper entitled:“Musiplectics: Computational Assessment of the Complexity of MusicScores” by Ethan Holder, Eli Tilevich, and Amy Gillick, publishedOctober 2015 in ONWARD '15 [ACM 978-1-4503-1995-9/13/10,http://dx.doi.org/10.1145/2508075.2514879], in a paper entitled: “AMethod for Measuring the Difficulty of Music Scores” by Yang-Eui Songand Yong Kyu Lee [www.ksci.re.krhttp://dx.doi.org/10.9708/jksci.2016.21.4.039] published April 2016 inthe Journal of The Korea Society of Computer and Information Vol. 21 No.4, or on available commercial application for writing music or songs areavailable, such as Ludwig (http://www.write-music.com/), and AnthemScore(https://www.lunaverus.com/), which are all incorporated in theirentirety for all purposes as if fully set forth herein.

The musical pieces database or table 89 shown as part of the arrangement80 b in FIG. 8 b was exampled as storing a single version or variant foreach of the stored physical pieces 88, such as the original version ofthe piece, for example as composed or created by a respective compositoror creator (referred to as version ‘5’ in the flow chart 70 in the FIG.7 ). However, the original version may be too complex or difficult to beplayed by a novice or learning player. In one example, it may be anadvantage to transcribe or adapt the musical piece to better adapt orpersonalize the musical piece to fit to the user skill level or playingabilities. In one example, he “Update Skill Level” step 102 may includesetting a pace or tempo adapted to fit the skill level value of the user36. Alternatively, or in addition, to adapting the pace or tempo to thespecific skill level of the user 36, the sequence of musical symbolsthat are presented to the user 36, may be transcribed to fit thespecific skill level of the user 36.

An improved musical pieces-table or database 89 b, that is based on thetable 89 in FIG. 8 b is described as part of a view 140 shown in FIG. 14. In one example, one of, part of, or all of, the musical pieces 88stored in the table 89 b, include multiple versions or variants, such asmodified arrangements, in addition to the original version (referred toas version ‘5’ in the flow chart 70 in the FIG. 7 ). Such version orvariants may be a simplified complexity or difficulty versions, such asthe version 4, 3, 2, and ‘1’ described in the flow chart 70 in the FIG.7 .

In the example shown in the view 140 in FIG. 14 , the table 89 b storesfive versions or variants of the musical Piece #4 88 d, each version 141a is associated with a skill level value 141 b, as described in a toprow 142. In addition to the original version 142 a, designated asversion ‘A’ or level ‘5’, that may be stored in the table 89 b as partof the “Store Version ‘5’” step 74, the table 89 b may comprise a leastsimplified version 142 b, designated as version ‘B’ or level ‘4’, thatmay be stored in the table 89 b as part of the “Store Version ‘4’” step74 a, a more simplified version 142 c, designated as version ‘C’ orlevel ‘3’, that may be stored in the table 89 b as part of the “StoreVersion ‘3’” step 74 b, a simplified version 142 d, designated asversion ‘D’ or level ‘2’, that may be stored in the table 89 b as partof the “Store Version ‘2’” step 74 c, and a more simplified version 142e, designated as version ‘E’ or level ‘1’, that may be stored in thetable 89 b as part of the “Store Version ‘1’” step 74 d. While fivelevels are exampled in the view 140 for the Piece #4 88 d, any number ofversions or variants, each associated with a different complexity ordifficulty level, may be equally used, such as 2, 3, 4, 6, 7, 8, 9, 10,12, 15, 20, or more. Further, each of the pieces 88 in the table 89 bmay be associated with a different number of versions or variants. Forexample, the original version of the Piece #3 88 c is associated withlevel ‘3’, and thus may include, for example, three versions, associatedwith levels ‘3’, ‘2’, and ‘1’.

In one example, the table 89 b shown in FIG. 14 may be used for adaptinga version of a selected musical piece to a person skill level value,such as described in a flow chart 140 a shown in FIG. 14 a , that isbased on the flow chart 100 shown in FIG. 10 . As described above, themusical piece to be played is selected, by the system or by the user 36,as part of the “Determine Skill Level & Prepare Musical Piece” step 93.However, the actual original or simplified version, or the selectedmusical piece that is to be played, is selected as part of a “SelectVersion/Level” step 143, that uses the musical piece versions table 89b. The selection in the “Select Version/Level” step 143 may be based onselecting the version that best fits the user 36 skill level value, suchas selecting the simplified version associated with the skill levelvalue that is associated with the user 36. In one example, the user 36is determined to be User #1 84 a, and the Piece #4 88 d is selected, aspart of the “Determine Skill Level & Prepare Musical Piece” step 93.Since the User #1 84 a is associated in the table 86 shown in FIG. 8 bwith a skill level value of ‘1’, the corresponding level ‘1’ version E142 e is selected as part of the “Select Version/Level” step 143. Inanother example, the user 36 is determined to be User #2 84 b, that isassociated in the table 86 shown in FIG. 8 b with a skill level value of‘3’, and thus the corresponding level ‘3’ version C 142 c is selected aspart of the “Select Version/Level” step 143. Similarly, in case wherethe user 36 is determined to be User #3 84 c, that is associated in thetable 86 shown in FIG. 8 b with a skill level value of ‘2’, and thus thecorresponding level ‘2’ version D 142 d is selected as part of the“Select Version/Level” step 143, and in case where the user 36 isdetermined to be User #4 84 d, that is associated in the table 86 shownin FIG. 8 b with a skill level value of ‘5’, and thus the correspondingoriginal version (level ‘5’) version A 142 a is selected as part of the“Select Version/Level” step 143.

In the example described in the flow chart 140 a shown in FIG. 14 a , aversion of a selected musical piece is selected once, based on, oraccording to, the user 36 skill level, as part of the “SelectVersion/Level” step 143, before the initiation of the practice session.Alternatively or in addition, the adapting of the corresponding versionof the musical piece may be performed during the practice session. Forexample, when after playing part of a musical piece a skill level valueof the player is updated, for example, as part of the “Update Skilllevel” step 102, such update is reflected (preferably immediately) inthe version that is used, adapting to the updated user 36 skill levelvalue, as exampled in a flow chart 140 b shown in FIG. 14 b , that isbased on the flow chart 110 shown in FIG. 11 . As described in the flowchart 140 b shown in FIG. 14 b , after updating the skill level value ofthe player 36 as part of the “Update Skill level” step 102, as analternative to simple selection of the part to be played as part of the“Select Next Part” step 111, the next part is selected using a versionof the musical piece that is based on, or adapted to, the updated skilllevel value, as part of a “Select Next Part Version/Level” step 144.

In one example, the user 36 is User #2 84 b that is associated with askill level value of ‘2’, and the selected the Piece #4 88 d is selectedas part of the “Determine Skill Level & Prepare Musical Piece” step 93.It is further assumed that the playing is based on parts having equallengths of 10 seconds. As explained in the flow chart 140 a, the versionC 142 c that corresponds to a level of ‘3’ is selected as part of the“Select Version/Level” step 143. Assuming, for example, that afterplaying of the 30-40 seconds of the musical piece using the version C142 c (that corresponds to the level of ‘3’), the skill level value isupgraded to ‘4’ as part of the “Update Skill level” step 102. In such acase, the next part to be used is selected to be the version B 142 bthat is associated with a complexity or difficulty level of ‘4’, and thesymbols that are displayed to the user 36 as part of the “Display NextSymbol” step 96 during the next part of 40-50 seconds, are based on, oraccording to, the newly selected version B 142 b. Similarly, in case ofmultiple playing errors, the skill level value may be downgraded to ‘2’as part of the “Update Skill level” step 102. In such a case, the nextpart to be used is selected to be the version D 142 d that is associatedwith a complexity or difficulty level of ‘2’, and the symbols that aredisplayed to the user 36 as part of the “Display Next Symbol” step 96during the next part of 40-50 seconds, are based on, or according to,the newly selected version D 142 d.

The flow chart 140 a shown in FIG. 14 a exampled the using of storedmultiple simplified variants or versions of a musical piece (such as inthe table 89 b) that were formerly ‘off line’ created before the startof the flow chart 140 a, such as by performing the flowchart 70 shown inFIG. 7 . Alternatively or in addition, only one version (or few versionsor variants) may be stored, such as the table 89 shown in FIG. 8 b , anda simplified version or variant that adapt to the skill level value ofthe user 36 may be created from the stored version by applying areal-time ‘on-the-fly’ simplifying process, such as the “ModifyArrangement” step 73, as part of the practice session. For example, onlythe original version (designated as Version A in the flow chart 70 shownin FIG. 7 , may be stored, as described above regarding the table 89 inthe arrangement 80 b. An example of such a flow chart 140 c is shown inFIG. 14 c , and is based on the flow chart 140 a that is shown in FIG.14 a . As an alternative (or in addition) to selecting a pre-storedversion as part of the “Select Version/Level” step 143, a simplifiedversion is generated in real-time as part of the “Modify Arrangement”step 73. For example, in case the musical piece selected as part of the“Determine Skill Level & Prepare Musical Piece” step 93 is Piece #4 88 dthat is associated with a complexity level of ‘5’, and the user 36 isidentified as the User #2 84 b that is associated with a skill levelvalue of ‘3’, then after the musical piece is selected as part of the“Determine Skill Level & Prepare Musical Piece” step 93, as part of the“Modify Arrangement” step 73 the version ‘3’ is created, similar to the“Store Version ‘3’” step 74 b, and this simplified version is usedthroughout the playing practice.

The flow chart 140 b shown in FIG. 14 b exampled the using of storedmultiple simplified variants or versions of a musical piece (such as inthe table 89 b) that were formerly ‘off line’ created before the startof the flow chart 140 b, such as by performing the flowchart 70 shown inFIG. 7 . Alternatively or in addition, only one version (or few versionsor variants) may be stored, such as the table 89 shown in FIG. 8 b , anda simplified version or variant of the part to be played that adapt tothe skill level value of the user 36 may be created when moving frompart to part along the playing practice session, from the single storedversion by applying a real-time ‘on-the-fly’ simplifying process, suchas the “Modify Arrangement” step 73, as part of the practice session.For example, only the original version (designated as Version A in theflow chart 70 shown in FIG. 7 , may be stored, as described aboveregarding the table 89 in the arrangement 80 b. An example of such aflow chart 140 d is shown in FIG. 14 d , and is based on the flow chart140 b that is shown in FIG. 14 b . As an alternative (or in addition) toselecting a pre-stored version of part of the musical pierce as part ofthe “Select Next Part Version/Level” step 144, a simplified version isgenerated in real-time as part of the “Modify Arrangement” step 73. Forexample, in case the musical piece selected as part of the “DetermineSkill Level & Prepare Musical Piece” step 93 is Piece #4 88 d that isassociated with a complexity level of ‘5’, and during the playingsession, as part of the “Update Skill Level” step 102 the skill levelvalue 85 b of the user 36 is updated to a skill level of ‘2’, then aspart of the “Modify Arrangement” step 73 the version ‘2’ is created,similar to the “Store Version ‘2’” step 74 c, and this simplifiedversion is used throughout the playing practice of the next part of themusical piece. For example, a user 36 may start with a complexity levelof ‘4’ that is adapted to a pre-stored value in the table 86, and thefirst 20 second may be played with a simplified version that correspondsto the skill level value of ‘4’. In case the user 36 skill level valueis determined after this period to stay at level ‘4’, the samesimplified version will continue to be used in the practice session.Upon updating the user 36 skill level value after this part to level ‘5’(assuming to be the original version complexity level), the next partwill use the original level version (version ‘A’ or ‘5’) instead of thesimplified version. However, is case the user 36 skill level value isdetermined after this period to be lowered to level ‘3’, a level ‘3’simplified version is created and used in the following part of themusical piece.

While the arrangement 80 in FIG. 8 exampled the case of using the clientdevice 35 that communicates over the Internet 22 with a server device 23c that is used for both the processing and the storing of the usersdatabase 31 a and the pieces database 31 b, the methods herein mayequally be implemented where the processing and/or the databases storingare performed in the client device 35, and there is no need for anyserver device 23 a or for an Internet 22 networking. Such configurationallows for performing the methods or steps herein even when or whereInternet connection is not available, or where the using on an externaldevice, such as the server device 23 a, is not recommended or available.

Such an arrangement 150 is shown in FIG. 15 , illustrating a clientdevice 35 a, which may be similar, identical, or different from theclient device 35. The client device 35 a stores the pieces database 31b, and its internal processor 12 performs the required analysis, as wellas executing any of the methods or steps herein. A flow chart 150 ashown in FIG. 15 a , is based on the flow chart 110 shown in FIG. 11 ,but adapted to be executed solely by the client device 35 a as part ofthe arrangement 150.

Many musical pieces are based on simultaneous playing of many musicalinstruments, and are composed to be played by a band or an orchestra. Auser experience of learning to play of a musical instrument becomes morerealistic and more enjoyable when the playing involves other musicalinstruments, either virtually where the sound of the other musicalinstruments is emulated by a system, or realistically when playingcooperatively with other users. In one example, the pieces database 31 bthat includes the table 89, further includes, for each of the storedmusical pieces, information regarding the various musical instrumentsinvolved in the actual playing of the respective musical piece.

An example of a table 89 a that is part of the pieces database 31 b isillustrated in a view 160 shown in FIG. 16 . In addition to the dataassociated with each of the stored musical pieces described in the table89, the table 89 a in the pieces database 31 b further includes anensemble data an example of additional stored information for the Piece#3 88 c, as shown in a table 161 (or in any other data structure). Foreach musical instrument involved in the musical piece, the table 161includes, as shown in a top row 162, a name or identifier 163 a of themusical instrument, a sheet music 163 b transcribed and adapted for thespecific musical instrument, and an audio file 163 c that includes theplaying of the symbols that are associated with the specific musicalinstrument when the musical piece is played. While the example of thetable 161 includes 6 musical instruments for playing the Piece #3 88 c,any number of musical instruments may be equally used. While the audiofiles 163 c are shown in the table 161 being of ‘.mp3’, ‘.wav’, ‘.xml’,‘.mid’, or ‘.aac’ types, any other format for music or audio may beequally used.

In the example of the table 161, the Piece #3 88 c is composed ofplaying a guitar 162 a that is associated with guitar sheet music fileGuitar-sheet.pdf and an audio file guitar.mp3, drums 162 b that areassociated with drums sheet music file Drums-sheet.doc and an audio filedrums.mp3, a piano 162 c that is associated with guitar sheet music filePiano-sheet.docx and an audio file Piano.wav, a bass 162 d that isassociated with bass sheet music file Bass-sheet,pdf and an audio fileBass.wav, a vocal 162 e that is associated with vocal sheet music fileVocal-sheet.doc and an audio file Vocal.aac, and a flute 162 f that isassociated with vocal sheet music file Flute-sheet.docx and an audiofile Flute.aac.

For any practicing session, the user 36 may defines the session as onethat includes only the specific musical instrument, or the interest ofhearing other musical instruments in addition to its own played musicalinstrument. Such definition may use a practicing table, such as practicetables 160 a and 160 b shown in FIG. 16 a , having a column ‘Source’ 164that associates with each of the musical instruments 162 a value ‘Self’denoting that this is the musical instrument to be practiced by the user36, a value ‘None’ in case the specific musical instrument is not to beinvolved in the practicing, a value ‘System’ if the user 36 isinterested is simulating the playing of this instrument by the system,and a value ‘User #X’ for hearing the playing of a specific other userthat cooperates in the practicing session.

In the practice tables 160 a or 160 b, the user 36 defines theparticipation and role of each instrument, such as the guitar 162 a, thedrums 162 b, the piano 162 c, the bass 162 d, the vocal 162 e, and theflute 162 f. In the example of the practice table 160 a (or in any otherdata structure). the user 36 defines the piano as the practicing device(value of ‘Self’), and all other instrument as defined as ‘None’. Hence,only the piano is involved in the practice, and the user 36 will nothear any other instruments. In the example of the practice table 160 b,the user 36 defines the Flute 162 f as the practicing device (value of‘Self’), the piano 162 c is not participating in the practice (value of‘None’), the guitar 162 a and the vocal 162 e are vocalized and emulatedby the system and heard by the user 36 (value of ‘System’), such as viathe speaker 78, the drums 162 b and the bass 162 d are both playedtogether with respective cooperative users User #1 84 a and User #3 84c, where both players are heard via the speaker 78.

An example of joint practice of three players acting as a band ororchestra is pictorially illustrated in a view 185 shown in FIG. 18 .The first player 36 is practicing playing of the piano 83 and uses theclient device 35, the second player 36 a is practicing playing of aflute 181 and uses a client device (shown as a tablet) 35 a placed on astand 183, and the third player 36 b is practicing playing of a guitar182 and uses a client device 35 b (shown as a tablet) placed on a stand183 a. The three players 36, 36 a, and 36 b are in a same location 184,where they can hear each other playing and each player can hear thesound emitted by all of the tablets. For example, the three players 36,36 a, and 36 b may be in the same room, in the same apartment, or in thesame building. While exampled in the arrangement 185 for three players,the systems and methods herein equally apply to any other number ofplayers, such as 2, 4, 5, 6, 7, 8, 9, 10, 12, 15, 18, 20, or moreplayers. The musical instruments may be different from each other.Alternatively, while different musical instruments are exampled in thearrangement 185, two or more of the musical instruments may be identicalor similar.

Further, one of, each of two of, each of most of, or each of, themusical instruments may be a Soprano instrument, such as flute, violin,soprano saxophone, trumpet, clarinet, oboe, and piccolo. Alternativelyor in addition, one of, each of two of, each of most of, or each of, themusical instruments may be an Alto instrument, such as alto saxophone,French horn, English horn, viola, and alto horn. Alternatively or inaddition, one of, each of two of, each of most of, or each of, themusical instruments may be a Tenor instrument, such as trombone,tenoroon, tenor saxophone, tenor violin, guitar, and tenor drum.Alternatively or in addition, one of, each of two of, each of most of,or each of, the musical instruments may be a Baritone instrument, suchas bassoon, baritone saxophone, bass clarinet, cello, baritone horn, andeuphonium. Alternatively or in addition, one of, each of two of, each ofmost of, or each of, the musical instruments may be a Bass instrument,such as double bass, bass guitar, contrabassoon, bass saxophone, tuba,and bass drum. Alternatively or in addition, one of, each of two of,each of most of, or each of, the musical instruments may be a stringinstrument, a woodwind instrument, a brass instrument, or a percussioninstrument.

A first practice table 180 (or any other data structure) that may bedefined by the first user 36 may include a source definition 164, asecond practice table 180 a that may be defined by the second user 36 amay include a source definition 164 a, and a third practice table 180 bthat may be defined by the third user 36 b may include a sourcedefinition 164 b, are shown in FIG. 18 a . Since the user 36 practicesthe piano 83, the piano row 162 c is associated with ‘Self’. The guitarrow 162 a is defined as ‘None’ since the user 36 is able to locallydirectly hear the guitar 182 played by the user 36 b, and similarly theflute row 162 f is defined as ‘None’ since the user 36 is able tolocally directly hear the flute 181 played by the user 36 a. The user 36may prefer to hear drums played remotely by the User #4 84 d, as definedby the ‘User #4’ on the drums row 162 b. Further, the user 36 may preferto not hear via the speaker 78 in the tablet 35 the bass and vocal, asdefined by the ‘None’ value in the respective rows 162 d and 162 e.

Since the user 36 a that defines the practice table 180 a practices theflute 181, the flute row 162 f is associated with ‘Self’. The guitar row162 a is defined as ‘None’ since the user 36 a is able to locallydirectly hear the guitar 182 played by the user 36 b, and similarly thepiano row 162 c is defined as ‘None’ since the user 36 a is able tolocally directly hear the piano 83 played by the user 36. The user 36 amay prefer to hear the bass played remotely by User #1 84 a, as definedby ‘User #1’ on the bass row 162 d. Further, the user 36 a may prefer tonot hear via the speaker 78 in the tablet 35 a the drums and vocal, asdefined by the ‘None’ value in the respective rows 162 b and 162 e. Forexample, the user 36 a may hear the drums via the tablet 35 used by theuser 36 that sources the sound from the User #4 84 d, as shown in thepractice table 180.

Similarly, since the user 36 b that defines the practice table 180 bpractices the guitar 182, the guitar row 162 a is associated with‘Self’. The flute row 162 f is defined as ‘None’ since the user 36 b isable to locally directly hear the flute 181 played by the user 36 a, andsimilarly the piano row 162 c is defined as ‘None’ since the user 36 bis able to locally directly hear the piano 83 played by the user 36. Theuser 36 b may prefer to hear the vocal played artificially by the systemas defined by ‘System’ on the vocal row 162 e. It may be assumed thatthe created vocal sound is heard by the users 36 and 36 b from a speaker78 in the tablet 35 b.

Further, the user 36 b may prefer to not hear via the speaker 78 in thetablet 35 b the drums and bass, as defined by the ‘None’ value in therespective rows 162 b and 162 d. For example, the user 36 b may hear thedrums via the tablet 35 used by the user 36 that sources the sound fromthe User #4 84 d, as shown in the practice table 180, and the bass viathe tablet 35 a used by the user 36 a that sources the sound from theUser #1 84 a, as shown in the practice table 180 a.

An example of a flow chart 170 for use by the server device 23 a and anyof the client devices (such as the client device 35 used by the firstplayer 36, the client device 35 a used by the second player 36 a, or theclient device 35 b used by the third player 36 b), for a case of jointpractice (such as shown in the arrangement 180) is shown in FIG. 17 ,and is based on the flow chart 110 shown in FIG. 11 .

The respective user defines a practice table as part of an “ObtainPractice Table” step 171, such as by using an input component 38 of therespective client device 35. For example, the first user 36 may definethe practice table 180, the second user 36 a may define the practicetable 180 a, and the third user 36 b may define the practice table 180b. The defined practice table is then sent, as part of a “Send Practicetable” step 172, from the respective client device, such as the clientdevice 35 used by the first player 36, the client device 35 a used bythe second player 36 a, or the client device 35 b used by the thirdplayer 36 b, to the server device 23 a, where it is received as part ofa “Receive Practice Table” step 172 a.

The symbols used, which are identified as part of an “Identify NextSymbol” 103 a, sent as part of the “Send Next Symbol” 95, and displayedas part of the “Display Next Symbol” step 96, are selected from thetable 161 based on the identification of ‘Self’ in the respectivepractice table. For example, for the first user 36, since the piano isdefined as ‘Self’ on the row 162 c in the respective practice table 180,the symbols that are displayed on the client device 35 are based on thePiano-sheet.docx based on the piano row 162 c in the table 161.Similarly for the second user 36 a, since the flute is defined as ‘Self’on the row 162 f in the respective practice table 180 a, the symbolsthat are displayed on the client device 35 a are based on theFlute-sheet.docx based on the flute row 162 f in the table 161.Similarly, for the third user 36 b, since the guitar is defined as‘Self’ on the row 162 a in the respective practice table 180 b, thesymbols that are displayed on the client device 35 b are based on theGuitar-sheet.pdf based on the guitar row 162 a in the table 161.

In a case where the user defines a ‘System’ as a source in a respectivepractice table, such as the vocal instrument 162 e in the practice table180 b used by the user 36 b, then the corresponding symbols of therespective instrument are identified as part of the “Identify NextSymbol” 103 a. In the example of the vocal instrument 162 e in thepractice table 180 b used by the user 36 b, the symbols from the fileVocal-sheet.doc of the vocal row 162 e of the table 161 are identified,and are sent by the server device 23 a to the client device 35 b as partof a “Send Other Symbols” step 174, and are received at the clientdevice 35 b as part of the “Receive Other Symbols” step 174 a. In thecase where few instruments are defined as ‘System’, then all the symbolsassociated with these musical instruments are identified as part of the“Identify Next Symbol” 103 a, sent by the server device 23 a to theclient device 35 b as part of a “Send Others Symbols” step 174, and arereceived at the client device 35 b as part of the “Receive OthersSymbols” step 174 a.

The received symbols are converted to a sound signal as part of a“Vocalize Symbols” step 175, and are played by the respective speaker 78of the client device 35 b as part of an “Emit Others Sound” step 177,thus simulating the presence of a vocal instrument.

In a case where the user defines a ‘User #X’ as a source in a respectivepractice table, such as the ‘User #1’ value in the bass row 162 d in thepractice table 180 a used by the user 36 a, then the respective soundfrom the defined user #1 84 a is sent from the server device 23 a by theclient device 35 a as part of a “Send Others Sounds” step 173, and isreceived by the client device 35 a as part of a “Receives Others Sounds”step 173 a. The received sound is then played by the respective speaker78 of the client device 35 b as part of the “Emit Others Sound” step177, thus simulating the local presence of a remote user. In the casewhere few instruments are defined as ‘User #X’ as a source, then all thesound signals received from these users are combined and sent from theserver device 23 a by the client device 35 a as part of a “Send OthersSounds” step 173, are received by the client device 35 a as part of a“Receives Others Sounds” step 173 a, and are then sounded by therespective speaker 78 of the client device 35 b as part of the “EmitOthers Sound” step 177.

It is further noted that the captured sound by a microphone 82 of therespective client device is analyzed as described above. However, suchanalysis as part of a “Analyze Sound” step 104 a needs to take intoaccount that the captured sound by the microphone 82 includes the soundemitted as part of the “Emit Others Sound” step 177 in addition to thesound emitted by the musical instrument that is part of the practicesession, and thus this emitted background sound that is emitted as partof the “Emit Others Sound” step 177 needs to be reduced from thecaptured sound, such as by any noise cancellation technique, to allowproper analysis of the sound of interest that is the result of playingthe musical instrument based on the displayed symbols as part of the“Display Next Symbol” step 96.

The flow chart 170 shown in FIG. 17 describes handling a ‘System’definition of a musical instrument by sending musical symbols of theselected musical instrument, and simulating the presence of the selectedmusical instrument by converting the symbols into sound that is playedon the respective speaker 78 of the client device 35 as part of an “EmitOthers Sound” step 177. Alternatively or in addition, the audio files163 c in the table 161, that may be recorded sound or an artificialsound after vocalizing, using any techniques that may be used in the“Vocalize Symbols” step 175. In such a case, described in a flowchart170 a shown in FIG. 17 a , the sound file of the selected musicalinstrument, or of multiple musical instruments if more than one isselected, is sent from the server device 23 a to the client device 35 aspart of a “Send Stored Sounds” step 176, is received by the clientdevice 35 as part of a “Receive Stored Sounds” step 176 a, to bedirectly sounded by the speaker 78 of the respective client device 35 aspart of a “Emit Others Sound” step 177 a.

In the case of multiple users practicing in concert, the server device23 a needs to synchronize the users so that the symbols displayed aspart of the “Display Next Symbol” step 96, in all the client devices ofall users are all align according to the musical piece, and aredisplayed at the same pace or tempo. The synchronization is performed aspart of a “Synchronize” step 178, that may comprise selecting a pace ortempo for all participants in the practice session, such as selectingthe lower pace or tempo in order to accommodate the lower skill levelparticipant. Further, the starting of the progress should be shared byall participants. For example, the flow may start as part of the“Synchronize” step 178 only after the last defined participant logged inand provided both the respective User ID as part of the “Receive UserID” step 92 a and the respective practice table as part of the “ReceivePractice Table” step 172 a.

The user which operates the client device that is executing the flowchart 170 may be selected as a source for sound by other users. Forexample, the user may be identified as User #1 84 a, which is selectedas the background bass source as the user 36 a that operates the clientdevice 35 a, as described in the respective practice table 180 a at thebass row 162 d. In such a case, the sound received from the user as partof the “Receive Sound” step 98 a is transmitted to the requesting user36 a as part a “Transmit Sound” step 179.

In the example of the arrangement 185 shown in FIG. 18 , the studentsare located in the same location 184, where they can each (and maybesee) each other. Alternatively or in addition, the students may belocated in different places, where they cannot hear each other, such asin different apartments, buildings, streets, cities, or counties,states, or countries. Such an arrangement 195 is shown in FIG. 19 ,illustrating the first player 36 that is located at a first location 184a is practicing playing of the piano 83 and uses the client device 35,the second player 36 a that is located at a second location 184 b ispracticing playing of a flute 181 and uses a client device (shown as atablet) 35 a placed on a stand 183, and the third player 36 b that islocated at a third location 184 c is practicing playing of a guitar 182and uses a client device 35 b (shown as a tablet) placed on a stand 183a.

In one scenario, the three players 36, 36 a, and 36 b may decide toindividually practice their respective musical instrument, without anybackground sound or any participation of any other instruments. Anexample of respective practice tables, such as a first practice table190 used by the first user 36, a second practice table 190 a used by thesecond user 36 a, and a third practice table 190 b used by the thirduser 36 b, are shown in FIG. 19 a . The first practice table 190 definesonly the piano as a participating musical instrument, shown as ‘Self’ inthe piano row 162 c, corresponding to the piano 83, the second practicetable 190 a defines only the flute as a participating musicalinstrument, shown as ‘Self’ in the flute row 162 f, corresponding to theflute 181, and the third practice table 190 b defines only the guitar asa participating musical instrument, shown as ‘Self’ in the guitar row162 a, corresponding to the guitar 182.

In such a scenario the “Send Others Sound” step 173 and thecorresponding “Receive Others Sounds” step 173 a, as well as the “SendOthers Symbols” step 174 and the corresponding “Receive Others Symbols”step 174 a of the flow chart 170 are not active, and no sound is emittedfrom the speaker 78 of any of the respective client devices 35, 35 a,and 35 b, as part of the “Emit Others Sound” step 177 a. Similarly, nosound is transmitted as part of the “Transmit Sound” step 179.

Alternatively or in addition, one, two, or all of the players may decideto use an artificial background sound of other musical instruments. Suchscenario is illustrated in an arrangement 205 shown in FIG. 20 . Theplayer 36 a decides not to use such capability and not to hear anybackground or additional musical instruments, thus no sound is emittedfrom the speaker 78 of any of the respective client device 35 a.However, the users 36 and 36 b may select to hear artificial backgroundsound of other musical instruments. A system generated sound of one ormore other instruments is shown carried over a data path 201 a to thelocation 184 a to be received by the tablet 35, and similarly a systemgenerated sound of one or more other instruments is shown carried over adata path 201 b to the location 184 c to be received by the tablet 35 b.

An example of respective practice tables, such as a first practice table200 used by the first user 36, a second practice table 200 a used by thesecond user 36 a, and a third practice table 200 b used by the thirduser 36 b, are shown in FIG. 20 a . Similar to the arrangement 195, thesecond practice table 200 a defines only the flute as a participatingmusical instrument, shown as ‘Self’ in the flute row 162 f,corresponding to the flute 181.

In addition to defining the piano as a participating musical instrument,the first practice table 200 defines a guitar as an artificialaccompanying musical instrument as shown in the guitar row 162 a‘System’, indicating that the data path 201 a carries symbols or soundthat is associated with guitar playing. The data path 201 a maycorrespond to the “Send Others Symbols” step 174 and the corresponding“Receive Others Symbols” step 174 a of the flow chart 170, where guitarrelated symbols, such as based on the Guitar-sheet.pdf as part of theguitar row 162 a in the table 161, are transferred to the client device35, and are then converted to sound as part of the “Vocalize Symbols”step 175, and then sounded from the speaker 78 of the respective clientdevice 35. As an alternative to sending symbols, the server 23 a maysend sound files as part of a “Send Stored Sounds” step 176 in the flowchart 170 a, such as based on the guitar.mp3 file as part of the guitarrow 162 a in the table 161, to be sounded by the speaker 78 of therespective client device 35.

Similarly, the third practice table 200 b defines both the piano and thevocal as an artificial accompanying musical instruments shown as‘System’ in the piano row 162 c and in the vocal row 162 e, indicatingthat the data path 201 b carries symbols or sound that is associatedwith combined piano and vocal guitar playing. The data path 201 b maysimilarly correspond to the “Send Others Symbols” step 174 and thecorresponding “Receive Others Symbols” step 174 a of the flow chart 170,where piano related symbols, such as based on the Piano-sheet.docx aspart of the piano row 162 c in the table 161, together with vocalrelated symbols, such as based on the Vocal-sheet.doc as part of thevocal row 162 e in the table 161, are transferred to the client device35 b, and are then converted to sound as part of the “Vocalize Symbols”step 175, and then sounded from the speaker 78 of the respective clientdevice 35 b. In this scenario, no sound is transmitted as part of the“Transmit Sound” step 179 by any of the client devices.

Alternatively or in addition, one, two, or all of the players may decideto use an actual sound by other practicing participants of other musicalinstruments. Such scenario is illustrated in an arrangement 215 shown inFIG. 21 . The first player 36 that is located at a first location 184 amay be identified as User #1 84 a, the second player 36 a that islocated at a second location 184 b may be identified as User #2 84 b,and the third player 36 b that is located at a third location 184 c maybe identified as User #3 84 c. In this example, the player 36 decidesnot to use such a capability and not to hear any background oradditional musical instruments, thus no sound is emitted from thespeaker 78 of any of the respective client device 35. However, the users36 a and 36 b may select to hear background sound of other musicalinstruments. In the example of the arrangement 215, the sound from thepiano 83 played by the user 36 is transmitted over a data path 211 a tothe server device 23 a, where it is retransmitted over a data path 211 bto the location 184 b to be sounded by the tablet 35 a to the user 36 a,and simultaneously over a data path 211 c to the location 184 c to besounded by the tablet 35 b to the user 36 b. Similarly, the sound fromthe flute 181 played by the user 36 a is transmitted over a data path212 a to the server device 23 a, where it is retransmitted over a datapath 212 b to the location 184 c to be sounded by the tablet 35 b to theuser 36 b.

An example of respective practice tables, such as a first practice table210 used by the first user 36, a second practice table 210 a used by thesecond user 36 a, and a third practice table 210 b used by the thirduser 36 b, are shown in FIG. 21 a . The first practice table 210 may beidentical to the first practice table 200, denoting that the player 36decides not to use the feature of hearing any background or additionalmusical instruments. The second practice table 210 a defines the fluteas a participating musical instrument, shown as ‘Self’ in the flute row162 f, corresponding to the flute 181. In addition, the piano row 162 cincludes the value of ‘User #1’, defining that the user 36 a isinterested in hearing the playing of the piano 83 by the user 36 at thelocation 184 a.

The third practice table 210 b defines the guitar as a participatingmusical instrument, shown as ‘Self’ in the guitar row 162 a,corresponding to the guitar 182. In addition, the piano row 162 cincludes the value of ‘User #1’, defining that the user 36 b isinterested in hearing the playing of the piano 83 by the user 36 at thelocation 184 a. Further, the flute row 162 f includes the value of ‘User#2’, defining that the user 36 b is interested in hearing the playing ofthe flute 181 by the user 36 a at the location 184 b.

The data path 211 a involves the transmitting of the captured sound bythe microphone 82 of the client device 35 to the server 23 a, and maycorrespond to the “Send Sound” step 98 and the respective “ReceiveSound” step 98 a. Similarly, the data path 212 a involves thetransmitting of the captured sound by the microphone 82 of the clientdevice 35 a to the server 23 a, and may similarly correspond to the maycorrespond to the “Send Sound” step 98 and the respective “ReceiveSound” step 98 a. The data path 211 b involves the transmitting as partof the “Transmit Sound” step 179 of the tablet 35 a of the soundcaptured by the tablet 35 that was received by the server device as partof the “Receive Sound” step 98 a. Similarly, the data paths 212 b and211 c respectively may involve the transmitting as part of the “TransmitSound” step 179 of the sound captured by the tablets 35 and 35 a thatwas received by the server device 23 a as part of the “Receive Sound”step 98 a.

In one example, all involved users are able to hear all other practicingparticipants of other musical instruments, providing a simulation orfeeling of a band or orchestra that play together at the same location.Such scenario is illustrated in an arrangement 225 shown in FIG. 22 . Inthis example, the sound captured by the client device 35 at the location184 a is carried over a data path 211 a to the server device 23 a, fromwhich this captured sound is transmitted to the client device 35 a atthe location 184 b over a data path 211 b and to the client device 35 bat the location 184 c over a data path 211 c.

Similarly, the sound captured by the client device 35 a at the location184 b is carried over a data path 212 a to the server device 23 a, fromwhich this captured sound is transmitted to the client device 35 at thelocation 184 a over a data path 212 c and to the client device 35 b atthe location 184 c over a data path 212 b, and the sound captured by theclient device 35 b at the location 184 c is carried over a data path 213a to the server device 23 a, from which this captured sound istransmitted to the client device 35 at the location 184 a over a datapath 213 b and to the client device 35 a at the location 184 b over adata path 213 c.

An example of respective practice tables, such as a first practice table220 used by the first user 36, a second practice table 220 a used by thesecond user 36 a, and a third practice table 220 b used by the thirduser 36 b, are shown in FIG. 22 a . The first practice table 220 definesthe flute as a participating musical instrument, shown as ‘User #2’ inthe flute row 162 f, corresponding to the flute 181. In addition, theguitar row 162 a includes the value of ‘User #3’, defining that the user36 is interested in hearing the playing of the guitar 182 by the user 36c at the location 184 c.

The second practice table 220 a defines the piano as a participatingmusical instrument, shown as ‘User #1’ in the piano row 162 c,corresponding to the piano 83. In addition, the guitar row 162 aincludes the value of ‘User #3’, defining that the user 36 a isinterested in hearing the playing of the guitar 182 by the user 36 c atthe location 184 c. The third practice table 220 b defines the piano asa participating musical instrument, shown as ‘User #1’ in the piano row162 c, corresponding to the piano 83. In addition, the flute is definedas a participating musical instrument, shown as ‘User #2’ in the fluterow 162 f, corresponding to the flute 181.

The example of the arrangement 180 involved multiple users at the samelocation, and the example of the arrangement 195 involved multiple usersat different locations. Any combination of one or more students in asingle location and one or more students in multiple locations mayequally be applied. Each user is “receiving” musical; symbols, such asnotes or chords, for their specific musical instrument that matchestheir respective playing level (and can be adapted in real time ‘on thefly’). Each user may configure the system to simultaneously hear thecomplete background of the musical piece, preferably minus the otherplayers he can hear as being in the same location, or virtually thesound produced in other locations over a fast and low-latency Internetconnection (such as by using 5G). For example, a single musical piececan be played together (jammed) by 5 people that are located in 3locations: 2 players are located at a home location practicing Guitar(for example at an entree level) and drums (for example at a mid-level),while they both hear the background music of the all involved musicalinstruments of the musical piece minus the guitar and drums, while aplayer at a work location, may simultaneously practice on a piano andhears all other players without (‘minus’) the piano sound, and 2 moreplayers may be located at a school location and may simultaneouslypractice Bass (for example at an advanced level) and vocal (for exampleat mid-level), with a background music that includes all other playersand instruments, but for example without guitar, drums, piano, bass, andvocals.

Each player may (e.g., synchronously) receive the musical symbols, suchas musical notes, that match his specific musical instrument, to bedisplayed for practicing, where the symbols or the pace of displayingthe symbols are adapted to the specific musical instrument and thespecific user skill level, which may be adapted in real-time during thepractice session. Further, each user may hear via his device therelevant background music and other players that may be locally orremotely located, using a low latency connection.

The table 161 shown in the view 160, the table 160 a and 160 b shown inthe FIG. 16 a , the tables 180, 180 a, and 180 b shown in FIG. 18 a ,the tables 190, 190 a, and 190 b shown in FIG. 19 a , the tables 200,200 a, and 200 b shown in FIG. 20 a , the tables 210, 210 a, and 210 bshown in FIG. 21 a , and the tables 220, 220 a, and 220 b shown in FIG.22 a , as well as the arrangement 185 shown in FIG. 18 , the arrangement195 shown in FIG. 19 , the arrangement 205 shown in FIG. 20 , and thearrangement 215 shown in FIG. 21 , all example the scenario of usingdifferent musical instruments, either as system generated sound or asbeing played by the cooperatively playing users. However, any twomusical devices shown, either as system generated sound or as beingcooperatively played by the users, may be identical or similar. Forexample, the user 36 a shown as playing the flute 181 in the arrangement185 shown in FIG. 18 , may equally play a piano, which may be identicalor similar to the piano 83 played by the user 36, or alternatively mayplay a guitar, which may be identical or similar to the guitar 182played by the user 36 b. Similarly, any system generated simulatedmusic, such as the system generated guitar 162 a shown as part of thetable 200 in FIG. 20 a , may be replaced with a system generated musicof a musical instrument that is identical to, or similar to, one that isactually played by one of the actual players. For example, the systemgenerated guitar 162 a shown as part of the table 200 in FIG. 20 a maybe replaced with a piano generated music, that may be identical to, orsimilar to, the piano 83 played by the user 36, may be replaced with aflute generated music, that may be identical to, or similar to, theflute 181 played by the user 36 a, or may be replaced with a guitargenerated music, that may be identical to, or similar to, the guitar 182played by the user 36 b.

An example of a software architecture 230 of software modules forperforming one of more of the flow charts, steps, or methods herein isdescribed in FIG. 23 , that may be part of the client device 35, part ofthe server device 23 a, or split between the two devices, so that theywork cooperatively. Each of the modules in the software architecture240, or any combination thereof, may be implemented as instruction inthe instructions block 37 a of the client device 35, and may use theservices of the operating system 37 b. Further, each of the modules inthe software architecture 240, or any combination thereof, may comprisesor may be used with a client-side API capable of providing web enabledfunction such as HTTP, for example using GET function, and may furtherbe implemented as JavaScript Object Notation (JSON) representationalstat transfer (REST) API. The exemplary arrangement 240 pictoriallydescribes an example of the process a user is experiencing from themoment the user wishes to play a certain music piece or selects a musicpiece out of a musical pieces or songs library. The process may betailor-made that music piece to the user 36 capability and focuses onthe user ability and learning process while playing. Finally, anadequate Background Music (BGM) is generated based on the user playingand possible jamming together with other players in the same space orfar away. The modules may be built for specific music instrument —andonce the user 36 changes the musical instrument, new settings areaccordingly configured.

An “User Playing Ability Analysis” 241 module is an optional module thatestimates the user playing ability based on previous performances,current progress in the application journey, and what proficiency theuser is expected to achieve. The user ability is a measurement of thecharacteristics of what the user played so far and the ability that theuser already knows—such as notes, chords, technique, rhythm, and more.The assessment can be a score or comparison of computed features usingAI learning from players at certain playing ability features.

A “Sheet Music Difficulty Assessment” 243 module is similar in computedfeatures to the “User Playing Ability Analysis” module 241 (derived fromthe sheet music), but estimates the difficulty of sheet music—by scoringvarious features including note/chord knowledge, rhythm, transitioncomplexity, tempo, and more. The scores include the average, minimum,maximum difficulty level and other statistical variants of a music piecearrangement.

Both of the “User Playing Ability Analysis” module 241 and “Sheet MusicDifficulty Assessment” module 243 feed a “Sheet Music Processing(Difficulty Levels)” module 242 which is the sheet music difficultylevel generation. The module 241 is the optional one, if no personaldata is taken a generalized difficulty level is created for “standard”users without personalization. The module 242 creates an arrangement fora given difficulty level—either based on the user knowledge or anaverage one. The arrangement is done based on rules, or AI from othermulti-level music pieces that are labeled. Each time a new arrangementhas created the difficulty of it is measured using the “Sheet MusicDifficulty Assessment” module 243. The result is one or morearrangements for the sheet music with various difficulty levels. Thedifferent difficulty levels arrangement can also be done either only bycrowdsourcing or a combination of automation and then improvement bycrowdsourcing and/or experts. The result of the “Sheet Music Processing(Difficulty Levels)” module 242 is an arrangement that is ready to fitthe user needs and based on the user abilities. The next stage for theuser is to start learning and eventually playing the music piece.

The playing is done via an “Presentation and Understanding User Play”module 205, using an application or utility. In an “Auto CreatedPractice Levels” 244 auto-generated practice levels are created based onthe arrangement and difficulty level. The way to create the practice isbased on learning from previously created sheet music with practicelevel or, by looking at the higher difficulty sequences and theirimportance in the arrangement and suggesting them as practice or, bylearning where other users fail at that difficulty level and suggestthem as practice. The practice can include educational videos—offline orlive on-demand with real teachers, rhythm training, learning of newnotes/chords or even AI/AR generated teaching videos.

Following preparation of the practice level the user can decide topractice, the user starts to play with the application or utility. Theapplication actual UI can have many forms (Visual UI, voiceinstructions, AR, etc.). While the user is playing the notes and chordsare presented to him along with score and the recognition qualityresults based on the recognition engine that comprises modules 249 to255, which are processing the user playing, removing background musicand output recognition and progress results.

While the user plays, the recognition engine in a “Recognition andSeparation” module 240 performs recognition that can result in a fewtypes of results, such as performing a “Process User Playing andHighlight or Ignore Parts” module 247,—OK recognition, NO recognition,error, and ignore (for repeated error). The “Process User Playing andHighlight or Ignore Parts” module 247 collects repeated sequences of badrecognition or ones the user is not able to play as expected. Once sucha sequence is considered as “problematic” it is saved for later practicesessions. From that moment on this sequence is not shown as “error”anymore but as “keep playing” or “ignore”. While the user plays thesequence again, the user also practices more and therefore can result ineventually playing that sequence in a satisfactory way. Once thishappens this sequence will be extracted out of the “problematic” list.In sequence we refer to any issue with the user playing including tempo,rhythm, technique, note/chord playing, transitions, etc.

In parallel during the user play—a “Process User Playing and ChangeDifficulty Level” module 248 processes the user's ability to play thesheet music at the current difficulty level—if the assessment is thatthe arrangement is an easy one to user an increased difficulty levelarrangement is presented and if the user fails too often decreaseddifficulty level is presented.

Once the playing ended the user is presented with score(s) and suggestedto take a personalized new practice session that is based on the user'sfinal difficulty level arrangement and the types of obstacles in playingthe user faced. This is done by an “Auto Generate Practice Level Basedon User Playing” module 246 that collects the sequences and other areasthe user fails in during the play. If the user wishes to play the sheetmusic again—the newly created practice is presented and if the userchooses to do it—it is set as the new practice level which is laterfollowed by the sheet musing playing with the adequate difficulty levelarrangement.

Modules 249 to 255 are the actual processes to enable the whole actualmusic learning of that music piece to happen. The application isenabling the user to play alone or jam together with others, whileplaying together with BGM that match the extra music to make it asong—such as for example adding drums, guitar, trumpet, bass, and vocalswhen the user plays piano. The user is expected to play together withBGM. As modern devices include multiple speakers and microphones that issome cases the application can select via which to play the BGM and viawhich to listen. This selection is done in order to assist therecognition engine and the separation process of the BGM from the actualplay.

The “Panning Microphone/Speaker Selection” module 249 is the basics fordecision out of the multiple microphones and speakers—which combinationyields the best experience for the user —hearing well the backgroundmusic and at the same being able to separate the music device from thetotal recorded sound (music device(s), BGM and added external noise).The idea is to auto-select the best configuration that can be alteredonline to maximize performance. The selection of the microphone andspeaker can influence the BGM preparation as part of a “BGM preparation”module 251—(music minus the played music device(s)) with matching to thedifficulty level —the less difficulty the more concrete BGM is needed ingeneral to compensate for minimal music played by the user. A “BGMManipulation—Volume/frequencies/Delete/Move Parts” module 254 performsadditional DSP on the BGM—such as lowering the volume, filters (such ashigh or low pass), removal or movement in time of some playing (such asin concurrent to the user playing), etc.

The modules 249, 251 and 254 are updated all the time based on therecognition and the separation assessment in real-time. The output ofthe BGM to be played is input to the recognition engine “Recognition andSeparation” module 250, in parallel to the module 255—playing that BGMthrough the selected speaker(s). The recording of the sound is performedin a “Record Microphone” module 253 via the selected microphone(s). Incase of joint practice (‘jamming’) a “Remove Other (e.g., Online)Players and/or music devices” module 252 is responsible for the input ofother musical instruments that play together with the user in the samespace and can be heard using the user microphones—those devices (userplaying) expected playing (result of the module 252 also input to therecognition engine in parallel to the expected BGM in the “BGMManipulation—Volume/frequencies/Delete/Move Parts” module 254.

An example of associating of the software architecture 230 of softwaremodules with the flow chart 150 a shown in FIG. 15 a is exampled as anarrangement 240 shown in FIG. 24 .

The arrangement 80 shown in FIG. 8 described an example where the user36 operates the musical instrument 83, and the generated sound by themusical instrument 83 in response to the user 36 action is captured bythe microphone 82 in the client device 35, and then analyzed formonitoring the user 36 playing. Alternatively or in addition, themusical device may be an electronic one, such as a MIDI controller, thatdirectly provides information regarding the user 36 action, thusallowing for directly monitoring the user 36 actions. Such anarrangement 150 b is shown in FIG. 15 b.

In the arrangement 150 b shown in FIG. 15 b , a MIDI controller 152 iswiredly connected to the client device 35′, using a cable 153 that isconnected between a connector 151 b on the MIDI controller 152 and amating connector 151 a on the client device 35′. In one example, thewired connection is MIDI based, where each of the connectors 151 a and151 b is a MIDI DIN-type connector, and the connecting cable 153 is aMIDI cable. Alternatively or in addition, the wired connection isUniversal Serial Bus (USB) based, where each of the connectors 151 a and151 b is a USB connector, and the connecting cable 153 is a USB cable.In both cases, the cable 153 provides a point-to-point connection andservers to carry serial, bytes-based messages (such as frames orpackets) respectively according to MIDI or USB protocol.

The actual actions of the user 36, such as the identification of theactual key that is pressed, the amount of pressure on the pressed key,knob turns, and slider changes are converted into MIDI data, and sentover the cable 153 from the MIDI controller 152 to the client device35′, which may then analyze the received MIDI data, or transmit the datato the server device 23 a for analysis. In such a case, the receivedactions data may be used to monitor the user 36 actions, as asupplement, or as an alternative, to the microphone 82. In one example,the MIDI controller 152 may comprise, or may consist of, a MIDIkeyboard.

As an alternative or in addition to the wired connection illustrated inthe arrangement 150 b shown in FIG. 15 b , a wireless connection may beused, as illustrated in an arrangement 150 c shown in FIG. 15 c . Thecable 153 and the mating connectors 151 a and 151 b may be replaced, orbe supplemented, with a wireless connection that uses the antenna 29 bin a MIDI controller 152 a and the antenna 29 a in a client device 35″.The antenna 29 a in a client device 35″ is connected to a wirelesstransceiver 28 a, that may be the same as, identical to, similar to, ordifferent from, the wireless transceiver 28 that is used communicationover the Internet 22 with the server device 23 a. In one example, thesame wireless communication interface is used for communicating withboth the server 23 a via the Internet 22 and with the wireless MIDIcontroller 152, and in such scenario the antenna 29 a is the same asantenna 29, and the wireless transceiver 28 is the same vas the wirelesstransceiver 28 a.

The wireless communication or network connecting the MIDI controller 152a and the client device 35″ may comprise a Wireless Personal AreaNetwork (WPAN), the wireless transceiver 28 a may comprise a WPANtransceiver, and each of the antennas 29 a and 29 b may comprise a WPANantenna. The WPAN may be according to, compatible with, or based on,Bluetooth™, Bluetooth Low Energy (BLE), or IEEE 802.15.1-2005 standards,or the WPAN may be a wireless control network that may be according to,or may be based on, Zigbee™, IEEE 802.15.4-2003, or Z-Wave™ standards.Alternatively or in addition, the wireless communication or networkconnecting the MIDI controller 152 a and the client device 35″ maycomprise a Wireless Local Area Network (WLAN), the wireless transceiver28 a may comprise a WLAN transceiver, and each of the antennas 29 a and29 b may comprise a WLAN antenna. The WLAN may be according to, may becompatible with, or may be based on, a standard selected from the groupconsisting of IEEE 802.11-2012, IEEE 802.11a, IEEE 802.11b, IEEE802.11g, IEEE 802.11n, and IEEE 802.11ac. Any wireless network hereinmay be over a licensed or unlicensed radio frequency band that may be anIndustrial, Scientific and Medical (ISM) radio band.

Alternatively or in addition, the wireless communication or networkconnecting the MIDI controller 152 a and the client device 35″ maycomprise a Wireless Wide Area Network (WWAN), the wireless transceiver28 a may comprise a WWAN transceiver, and each of the antennas 29 a and29 b may comprise a WWAN antenna. Any WWAN herein may be a wirelessbroadband network. The WWAN may be a WiMAX network, the antenna may be aWiMAX antenna and the wireless transceiver may be a WiMAX modem, and theWiMAX network may be according to, compatible with, or based on, IEEE802.16-2009. Alternatively or in addition, the WWAN may be a cellulartelephone network, the antenna may be a cellular antenna, and thewireless transceiver may be a cellular modem, where the cellulartelephone network may be a Third Generation (3G) network that may use aprotocol selected from the group consisting of UMTS W-CDMA, UMTS HSPA,UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, and GSM EDGE-Evolution, or thecellular telephone network may use a protocol selected from the groupconsisting of a Fifth Generation (5G) or Fourth Generation (4G) networkthat uses HSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be basedon IEEE 802.20-2008.

A flowchart 150 d shown in FIG. 15 d , which is based on the flowchart90 shown in FIG. 9 , describes the operation in case where the musicalinstrument is an electronic one that directly responds to the user 36actions. As an alternative to capturing the sound generated by themusical instrument in response to the user 36 action as part of the“Capture Sound” step 97, the client device receives from the electronicmusical instrument, such as the wired MIDI controller 152 of thearrangement 150 b or the wireless MIDI controller 152 a of thearrangement 150 c, as part of a “Capture Operation” step 154. The clientdevice 35′ receives a message that describes the user 36 operation fromthe MIDI controller 152 at the connector 151 a via the cable 153, andthe client device 35″ receives a message that describes the user 36operation from the wired MIDI controller 152 at the antenna 29 a usingthe wireless transceiver 28 a from the wireless MIDI controller 152 a.In one example, the received action as part of the “Receive Operation”step 154 is analyzed and processed in the client device 35.Alternatively or in addition, the client device 35 sends the receiveduser action as part of a “Send Operation” 155, and the server device 23a receives the sent user action as part of a “Receive Operation” step155 b, as an alternative to the “Receive Sound” step 98 a.

As an alternative to the “Analyze Sound” step 104, the received useraction is compared, as part of a “Compare Operation” step 156 with thecorresponding symbol (such as note or chord) that was displayed to theuser 36 as part of the “Display Next Symbol” step 96. Similarly, as analternative to the “Correct Sound ?” step 99, as part of a “CorrectAction” step 157, the received user action is determined to be correctis it corresponds to the displayed symbol.

Any music file herein, such as the files 87 d in the table 89 or thefiles 87 d in the table 89 a, may be according to an industry standardformat. In one example, these files may be according to, or based on,MusicXML standard, such as MusicXML 3.1 version, which is an XML-basedfile format for representing Western musical notation, developed by, andavailable from, the W3C Music Notation Community Group and released inDecember 2017. The MusicXML standard is designed for sharing sheet musicfiles between applications, and for archiving sheet music files for usein the future.

Alternatively or in addition, any music file herein, such as the files87 d in the table 89 or the files 87 d in the table 89 a, may beaccording to, or based on, ABC notation, which is a shorthand form ofmusical notation. In basic form it uses the letters A through G, letternotation, to represent the given notes, with other elements used toplace added value on these—sharp, flat, the length of the note, key,ornamentation. This form of notation began as an ASCII character setcode that could facilitate the sharing of music online and also added anew and simple language for software developers, not unlike othernotations designed for ease, such as tablature and solfège. ABC notationbeing ASCII-based, any text editor can be used to edit the code. Evenso, there are now many ABC notation software packages available thatoffer a wide variety of features, including the ability to read andprocess ABC notation into MIDI files and as standard “dotted” notation.

Alternatively or in addition, any music file herein, such as the files87 d in the table 89 or the files 87 d in the table 89 a, may beaccording to, or based on, DARMS File Format (also known as theFord-Columbia Format), GP*, which is Guitar Pro sheet music andtablature file, KERN, which is Kern File Format sheet music file, LY,which is LilyPond sheet music file, Music Encoding Initiative (MEI) fileformat that attempts to encode all musical notations, Finale sheet musicfile such as MUS or MUSX, MuseScore sheet music file such as MSCX orMSCZ, Standard Music Description Language (SMDL) sheet music file, orSibelius (SIB) sheet music file.

While the client device is exampled in the arrangement 30 ascommunicating with the server device 23 a via a wireless network 39using the wireless transceiver 28 and the antenna 29, a wiredcommunication may be equally used as an alternative or in addition tothe wireless connectivity. In wired communication, the antenna 29 isreplaced with a connector and the wireless transceiver 28 is replacedwith a wired transceiver. The wires communication may use any wirednetwork medium such as a single wire or two wires, and may comprise aShielded Twisted Pair (STP) or an Unshielded Twisted Pair (UTP).Alternatively or in addition, the network medium may comprise a LANcable that may be based on, or may be substantially according to,EIT/TIA-568 or EIA/TIA-570 standard, and may comprise UTP or STPtwisted-pairs, and the connector may be an RJ-45 type connector.Alternatively or in addition, the network medium may comprise an opticalcable and the connector may be an optical connector, and the opticalcable may comprises, may use, or may be based on, Plastic Optical Fibers(POF). Alternatively or in addition, the network medium may comprise ormay use a DC power carrying wires connected to a vehicle battery.

Any wired network herein may be a Personal Area Network (PAN), anyconnector herein may be a PAN connector, and any wired transceiverherein may be a PAN transceiver. Alternatively or in addition, any wirednetwork herein may be a Local Area Network (LAN) that may beEthernet-based, ant connector herein may be a LAN connector, and anytransceiver herein may be a LAN transceiver. The LAN may be accordingto, may be compatible with, or may be based on, IEEE 802.3-2008standard. Alternatively or in addition, the LAN may be according to, maybe compatible with, or may be based on, 10Base-T, 100Base-T, 100Base-TX,100Base-T2, 100Base-T4, 1000Base-T, 1000Base-TX, 10GBase-CX4, or10GBase-T; and the LAN connector may be an RJ-45 type connector.Alternatively or in addition, the LAN may be according to, may becompatible with, or may be based on, 10Base-FX, 100Base-SX, 100Base-BX,100Base-LX10, 1000Base-CX, 1000Base-SX, 1000Base-LX, 1000Base-LX10,1000Base-ZX, 1000Base-BX10, 10GBase-SR, 10GBase-LR, 10GBase-LRM,10GBase-ER, 10GBase-ZR, or 10GBase-LX4, and the LAN connector may be afiber-optic connector. Alternatively or in addition, any network hereinmay be a packet-based or switched-based Wide Area Network (WAN), anyconnector herein may be a WAN connector, and any transceiver herein maybe a WAN transceiver. Alternatively or in addition, any wired networkherein may be according to, may be compatible with, or may be based on,a Serial Peripheral Interface (SPI) bus or Inter-Integrated Circuit (PC)bus.

Any method herein may be in combination with an Augmented Reality (AR)system that may simulate a virtual environment to the person. Forexample, the display 81 of the client device 35 may be implemented, asan alternative or as addition to the display 81 described herein, as anHead-Mounted Display (HMD) that may be worn on the forehead, such as aharness or helmet-mounted, as a device resembling eyeglasses, as aHead-Up Display (HUD) that is a transparent display that presents datawithout requiring users to look away from their usual viewpoints, ascontact lenses, as Virtual Retinal Display (VRD) that is a personaldisplay device where a display is scanned directly onto the retina of aviewer's eye, or as part of a Spatial Augmented Reality (SAR) thataugments real-world objects and scenes, without the use of specialdisplays such as monitors, head-mounted displays or hand-held devices.

The interaction or notification with the user 36 is described hereinusing visual displaying by the display 81, such as in the “Display NextSymbol” step 96, as part of the “Display Error” step 96 a, as part ofthe “Display Feedback” step 96 b, and alternatively or in addition byusing audible sounding by the sounder 78, such as in the “VocalizeInstruction” step 114 or Vocalize Feedback” step 114 a. Alternatively orin addition to the visual or audible (or both) notifications to the user36, a haptic notification may be used. The haptic notification may use,or may be based on, cutaneous, kinaesthetic, orhaptic technologies. Thehaptic notification may be based on vibrations, that may be produced byan Eccentric Rotating Mass (ERM) actuator, a Linear Resonant Actuator(LRA), piezoelectric actuators, an unbalanced motor, a loudspeaker, anultrasound transducer, or an air vortex ring.

A human speech herein may be produced using a hardware, software, or anycombination thereof, of a speech synthesizer, which may beText-To-Speech (TTS) based. The speech synthesizer may be aconcatenative type, using unit selection, diphone synthesis, ordomain-specific synthesis. Alternatively or in addition, the speechsynthesizer may be a formant type, and may be based on articulatorysynthesis or Hidden Markov Models (HMM) based. Further, any speechsynthesizer herein may be based on, or may use, any of the schemes,techniques, technologies, or arrangements described in the bookentitled: “Development in Speech Synthesis”, by Mark Tatham andKatherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN:0-470-85538-X, in the book entitled: “Speech Synthesis and Recognition”by John Holmes and Wendy Holmes, 2^(nd) Edition, published 2001 ISBN:0-7484-0856-8, in the book entitled: “Techniques and Challenges inSpeech Synthesis—Final Report” by David Ferris [ELEC4840B] publishedApr. 11, 2016, or in the book entitled: “Text-to-Speech Synthesis” byPaul Taylor [ISBN 978-0-521-89927-7] published 2009 by CambridgeUniversity Press, which are all incorporated in their entirety for allpurposes as if fully set forth herein.

Any device, component, or apparatus herein, may be used with, integratedwith, or used in combination with, a Virtual Reality (VR) system thatsimulates a virtual environment to a person. The communication with theVR system may be wired or wireless, and the VR system may comprise aHead-Mounted Display (HMD). The simulated virtual environment.

Any apparatus herein, which may be any of the systems, devices, modules,or functionalities described herein, may be integrated with a smartphoneor a tablet. The integration may be by being enclosed in the samehousing, sharing a power source (such as a battery), using the sameprocessor, or any other integration functionality. In one example, thefunctionality of any apparatus herein, which may be any of the systems,devices, modules, or functionalities described here, is used to improve,to control, or otherwise be used by the smartphone. In one example, ameasured or calculated value by any of the systems, devices, modules, orfunctionalities described herein, is output to the smartphone device orfunctionality to be used therein. Alternatively or in addition, any ofthe systems, devices, modules, or functionalities described herein isused as a sensor for the smartphone device or functionality.

While any client device herein, such as the client device 35, isdescribed herein as being separate and distinct device from any musicalinstrument herein, such as the piano 83, they may equally be integratedwith each other. The integration of a client device with a musicalinstrument may involve sharing a component such as housing in the sameenclosure, sharing the same connector such as sharing a power connectorfor connecting to a power source, where the integration involves sharingthe same connector for being powered from the same power source. Theintegration with the appliance may involve sharing the same powersupply, sharing the same processor, or mounting onto the same surface.

Any device herein, such as each of the servers 23 a and 23 b, mayconsists of, may be part of, may comprises, or may be integrated with aserver, and may be storing, operating, or using, a server operatingsystem that may consist of, may comprise, or may be based on, one out ofMicrosoft Windows Server®, Linux, or UNIX. Alternatively or in addition,the server operating system may consist of, may comprise, or may bebased on, one out of Microsoft Windows Server® 2003 R2, 2008, 2008 R2,2012, or 2012 R2 variant, Linux™ or GNU/Linux-based Debian GNU/Linux,Debian GNU/kFreeBSD, Debian GNU/Hurd, Fedoram, Gentoo™, Linspire™,Mandriva, Red Hat® Linux, SuSE, and Ubuntu®, UNIX® variant Solaris™,AIX®, Mac™ OS X, FreeBSD®, OpenBSD, and NetBSD®.

The device 35, or any other device or apparatus herein, may be a clientdevice that may typically function as a client in the meaning ofclient/server architecture, commonly initiating requests for receivingservices, functionalities, and resources, from other devices (servers orclients). Each of the these devices may further employ, store,integrate, or operate a client-oriented (or end-point dedicated)operating system, such as Microsoft Windows® (including the variants:Windows 7, Windows XP, Windows 8, and Windows 8.1, available fromMicrosoft Corporation, headquartered in Redmond, Wash., U.S.A.), Linux,and Google Chrome OS available from Google Inc. headquartered inMountain View, Calif., U.S.A. Further, each of the these devices mayfurther employ, store, integrate, or operate a mobile operating systemsuch as Android (available from Google Inc. and includes variants suchas version 2.2 (Froyo), version 2.3 (Gingerbread), version 4.0 (IceCream Sandwich), Version 4.2 (Jelly Bean), and version 4.4 (KitKat),Android version 6.0 (Marshmallow), Android version 7.0 (Nougat), Androidversion 8.0 (Oreo), Android version 9.0 (Pie), Android 10, Android 11,iOS (available from Apple Inc., and includes variants such as versions3-7), Apple iOS version 8, Apple iOS version 9, Apple iOS version 10,Apple iOS version 11, Apple iOS version 12, Apple iOS version 13, AppleiOS version 14, Windows® Phone (available from Microsoft Corporation andincludes variants such as version 7, version 8, or version 9), orBlackberry® operating system (available from BlackBerry Ltd.,headquartered in Waterloo, Ontario, Canada). Alternatively or inaddition, each of the devices that are not denoted herein as a server,may equally function as a server in the meaning of client/serverarchitecture. Any Operating System (OS) herein, such as any server orclient operating system, may consists of, include, or be based on areal-time operating system (RTOS), such as FreeRTOS, SafeRTOS, QNX,VxWorks, or Micro-Controller Operating Systems (μC/OS).

The device 35, or any other client device or apparatus herein, may behoused in a single enclosure that is a hand-held enclosure, a portableenclosure, or a surface mountable enclosure. Further, the device 35, orany other client device or apparatus herein, may consist of, maycomprise, may be part of, or may be integrated with, a notebookcomputer, a laptop computer, a media player, a cellular telephone, atablet device, or a smartphone, such as a smartphone that consists of,comprises, or is based on, an Apple iPhone 12 or a Samsung Galaxy S20.

Any device, component, or apparatus herein, such as the device 35 or anyother client device or apparatus herein, may be structured as, may beshaped or configured to serve as, or may be integrated with, a wearabledevice. For example, any apparatus or device herein may be wearable onan organ such as on the person head, and the organ may be eye, ear,face, cheek, nose, mouth, lip, forehead, or chin. Alternatively or inaddition, any apparatus or device herein may be constructed to have aform substantially similar to, may be constructed to have a shapeallowing mounting or wearing identical or similar to, or may beconstructed to have a form to at least in part substitute for, headwear,eyewear, or earpiece. Any headwear herein may consist of, may bestructured as, or may comprise, a bonnet, a headband, a cap, a crown, afillet, a hair cover, a hat, a helmet, a hood, a mask, a turban, a veil,or a wig. Any eyewear herein may consist of, may be structured as, ormay comprise, glasses, sunglasses, a contact lens, a blindfold, or agoggle. Any earpiece herein may consist of, may be structured as, or maycomprise, a hearing aid, a headphone, a headset, or an earplug.Alternatively or in addition, any enclosure herein may be permanently orreleasably attachable to, or may be part of, a clothing piece of aperson. The attaching may use taping, gluing, pinning, enclosing,encapsulating, a pin, or a latch and hook clip, and the clothing piecemay be a top, bottom, or full-body underwear, or a headwear, a footwear,an accessory, an outwear, a suit, a dress, a skirt, or a top. Existingmethods and systems provide an application executed by a computingplatform or computing device such as a mobile phone, smart phone,tablet, a laptop, a desktop computer, or the like. The applicationenables a user to choose a piece of music and level from a selection ofmusical pieces and playing levels.

Background music (BGM) (also: audio playback) provided by theapplication may then be played through one or more speakers associatedwith the device or external thereto, and the user can play and/or singalong with the BGM, which provides the user with an improved musicexperience. The sound generated by the user through playing aninstrument and/or singing may herein also be referred to as“user-generated sound”. The user-generated sound combined with, forexample, the BGM, sound generated by other users playing an instrumentand/or signing, and/or environment noises, may herein be referred to as“composite sound”.

The user may also be provided with presented notes, tabs or othermusical notations displayed on a display device such as a displayassociated with the computing platform. The accompanying BGM is playedvia one or more of the device speakers, external speakers, headphones,and/or even via another device.

As the user plays along with the BGM, the audio may be captured by thedevice through one or more microphones, wherein the used microphone(s)can also be internal or external to the device. The method and systemcan monitor the sound and provide feedback to the user regarding theuser's playing, including for example indicating playing errors,suggesting a higher or lower playing level, lowering the volume of theBGM, or the like.

The system and method may have additional capabilities such assupporting jam sessions in which other users also play, wherein theother users are at the same location or at other locations from theuser. In further embodiments, the user can hear the playing of otherusers synthesized as expected, rather than as played, such that the useris not influenced by others' errors, and wherein each player can play inaccordance with his own schedule. The general operation of the system isdetailed in association with FIG. 25 below.

However, existing methods and systems have a number of drawbacks. Onesuch drawback relates to the deficient performance of such methods andsystems due to sub-optimal usage of the available I/O devices.

Currently available devices such as but not limited to smartphones,comprise a plurality of I/O devices, such as one or more microphones andone or more speakers. Some devices may comprise three microphones,located for example on the bottom, upper front, and/or upper back of thedevice, and three speakers, one on the top and two at the bottom of thedevice. The plurality of I/O devices are generally required for thevarious operation modes and algorithms executed by the device, such ashandset or speaker modes, video capture, noise reduction, echocancellation, keyword triggering, or the like.

The exact characteristics of the speakers and the microphones includinglocation, direction, frequency response, delayed playback, bandpassfilters, polar pattern, sound enhancements, and/or recording may beunknown or undocumented, and can change even with different hardwaresuppliers or operating system updates. Additionally, the hardware itselfcan be blocked due to the device placement, malfunction due to hardwareissues, cases, dirt, or the like.

By using a sub-optimal set of I/O devices the system and method canoffer only deficient user experience, such as poor balance, incorrectdetection of playing errors including false negative (missing playingerrors) or false positives (identifying correctly played notes aserroneous), or the like, which may result in the user being frustratedand not completing tasks such as playing selected pieces.

Thus, in accordance with some embodiments of the disclosure, there isprovided a method and system for selecting and configuring the I/Odevices for playing sound to the user and receiving sound from the user,in accordance with the I/O device availability, acoustic environment,supported and used algorithms, or the like. Further consideration mayinclude the possible device combination options. For example, in the iOSoperating system, an application can access only one microphone at atime, and in order to produce spatial audio the upper speaker can beplayed together with one of the lower speakers, wherein the selected onedepends on the device position.

The I/O device selection may be performed towards the specific target ofoptimizing the user's experience. This target may require separation ofthe user's playing from the rest of the captured audio including BGM,other users and external noises, recognizing the user's playing,enhancing the BGM and the other users if exist, and selecting andconfiguring the I/O devices according to any of the above. High qualityseparation may be required for enabling improved recognition of theuser's playing, i.e., transcribing the played notes or chords. Theseparation may be enabled, among others, because the system and methodhave prior knowledge about the played BGM (and the sound or expectedsound of other users), and because the BGM and other sounds are musicalinterfering signals rather than a random (e.g., spectrally homogenous)signal.

Once the sound is separated, updated enhancement of the BGM and updatedI/O device selection may be performed, resulting in improvedrecognition, sound experience, and hence better user experience andsatisfaction, which may be expressed in improved success in completingplaying tasks, providing a high rating of the application, recommendingthe application to friends, or the like.

For example, if the BGM is of significantly lower volume or soundintensity than the music played by the user, or in differentfrequencies, separation is quite straightforward. However, in manysituations there is overlap or similarity in the volume, sound intensityand/or frequencies. Additional sound may also be present which mayinclude, or be affected by the environmental noise, the location andorientation of the device wherein certain speakers or microphones can beblocked or capture external noises, the user's playing capabilities, theused music instrument, or the like.

The I/O device selection can take into account many factors, includingthe model of the used device, the specific device configuration, thepossible I/O device combinations, the played piece, the level, andothers. For example, in iPhone 12 devices, the upper front microphone islocated next to the upper speaker. Thus, for sound-to-noise ratio (SNR)reasons, it may be preferable to select the back microphone rather thanthe front one, and play BGM via the lower speaker. However, if thedevice has a cover which is close to the surface, the back microphonemight be blocked, in which case it may be preferable to play the BGMfrom the upper speaker and use the microphone on the lower side, or theother way around, i.e., play from the lower speaker and use the frontupper microphone. In another example, in devices including iPhone 12devices, the device's internal audio processing makes the device playthrough the upper and lower speakers simultaneously. The signal playedby the top speakers may saturate the upper microphone, therefore thelower microphone may be preferred. In another example, the upper speakermay not be suitable as the main playback speaker as it distorts theaudio when driven at high amplitudes. In yet another example, the backmicrophone is empirically shown to provide better results.

It will be appreciated that the I/O device selection can vary over timeas the user plays within the same session or in different sessions,depending on the recognition engine and results, the played BGM, theenvironmental noises, device orientation relative to the one or moreplayers, and more.

In some embodiments, one or more audio processing algorithms can beapplied to any components of the separated sound, such as volumereduction and/or compressing certain frequencies from the BGM and otherexternal digitally produced sounds, which that can interfere with theuser's expected playing.

A typical scenario in a music teaching application in accordance withthe disclosure may thus be described as follows: a user may select aninstrument to play, such as a guitar or a piano and a piece to play. TheBGM may be played via one or more speakers, wherein the music, includingthe BGM and what the user is playing is recorded via one or more of themicrophones. The selected speakers and microphones may be based on theavailable I/O devices, the position of the device within a room, roomacoustics, the position of the device relative to the player, andpossibly additional factors. The I/O devices are selected to yield aninput for the separation and recognition process, such that the user'splaying can be recognized and monitored, while maintaining a userexperience that meets at least one sound quality criterion, for example,by silencing the BGM as little as possible, eliminating recognitioninterference, providing high Signal to Noise (SNR) ratio, thus enablinga user to play and recognize accurately, complete stages, etc.

The I/O device selection may be updated intra-session or inter-sessionto provide good separation and thus good recognition and feedback, aswell as good quality BGM which improves the user experience, e.g., asdefined by the at least one sound quality criterion. The separation andrecognition may be performed using hardware and/or software audioprocessing algorithms such as echo cancellation, internal deviceacoustic echo cancellation (AEC), sound compression, source separation,and/or the like. Following recognition of the user's playing, the usermay be provided with feedback indicating how well he played the expectednotes or chords.

In some embodiments, additional sensors (inertial and/or non-inertial)may be used, such as cameras, linear acceleration sensors, angularacceleration sensors, gyroscopes, in order to identify a position and/ororientation of the device, the distance from the user and from themusical instrument, the type of music instrument being played by theplayer or players, and/or the like. In some embodiments, recognition maybe performed by fusing visual and audio sources. For example, the backcamera can identify that some obstacle covers a microphone.

The disclosed system and method thus provide for selecting andconfiguring the input devices through which a user learning to play iscaptured, and the output devices to be used for playing to the user theBGM and other roles of the arrangement in which the user isparticipating. The selections and configurations are such that on onehand separating the user's sound from the BGM, other users andbackground noise is better than in other selections or configurations,thereby providing for better monitoring of the user's playing.Furthermore, the selections or configurations improve the user'sexperience, in terms for example of the balance, interference, and othersound quality parameters, relative to other selections orconfigurations.

Reference is now made to FIG. 25 . A user 25100 is learning to playguitar 25104. User 25100 may activate an application installed on orexecuted by device 25108. Device 25108 may be a tablet computer, smartphone, a mobile phone, a desktop computer, a laptop computer or thelike. User 25100 may select a song or another piece to learn, andoptionally a difficulty level or player level.

User 25100 is then provided with musical instructions such as notes25112 of the selected piece, and sound 25116 containing the BGM for theselected piece is played through one or more speakers of device 25108such as speaker 25114, or another device such as earphones, externalspeaker, or the like. Musical instructions may also pertain toexpression and/or temp including, for example, “accelerando”, “adagio”,“crescendo”, “piano”, “pizzicato”, etc. Optionally, the roles of otherusers playing in the arrangement of the selected piece may also beplayed by the selected speaker. In some embodiments, the roles areplayed by the other users, while in other embodiments, sound 25116comprises the expected roles of the other users. As the BGM is played,cursor 25120 may be displayed, showing user 25100 the current note to beplayed.

The user's playing, as well as the sound 25116 and optionally additionalsound, such as environmental noises, are captured by one or moremicrophones of device 25108 such as microphone 25118 and/or externalmicrophones, to generate processable audio-data for analysis.

In some embodiments, device 25108 may execute a standalone application,in which the BGM is obtained and provided by the application, andanalysis of the user's playing is also performed by the application. Inother embodiments, the application may be a client application,communicating with a corresponding server application. In thisconfiguration, device 25108 may be in wired or wireless communicationwith server 25128, through channel 25124, such as the Internet,intranet, LAN, WAN, 5G, or the like. In such case, the music offering,the BGM generation and the analysis may be performed by either theclient application, the server application, or a combination thereof. Itwill be appreciated that if further users are playing, device 25108needs to be connected to a server or to another client application inorder to communicate with the device used by the other user.

When user 25100 has finished playing the piece, user 25100 may bepresented with the analysis results, comprising for example playingerrors, general comments, music type or level change recommendations, orthe like.

Referring now to FIG. 26 , showing a flowchart of steps in a method forteaching playing a musical instrument, in accordance with someembodiments of the disclosure. The flowchart depicts the steps taken byan application or program since a user starts the application or programand selects a piece to play, wherein the process may be tailor-made tothe user's selections, specific musical instrument and/or abilities.

Step 26204 (“USER PLAYING LEVEL ANALYSIS”) may for example include: theuser's playing level may be estimated. Estimation may be performed, forexample, by letting the user select a level, by asking the user to playa piece, assessing the user's performance and presenting the user withmore and more difficult pieces until his performance is below a certainthreshold, for example he has more than a predetermined number of errorsper minute on average. Further estimation may be performed by retrievingestimation results or playing results of previous sessions, by theuser's answers to theoretical questions related to notes, chords,technique, rhythm, or the like. The estimation may also be based onartificial Intelligence (AI) techniques, such as Neural Networks (NN)used for assessing the user's level by comparison to known levels ofother users.

Step 26208 (“Sheet music difficulty analysis and/or assessment”) may forexample include: the difficulty level of the piece the user selected maybe assessed, based for example on scoring various features of the pieceor the arrangement, including note or chords, rhythm, transitioncomplexity, tempo, and more. The difficulty level may include anaverage, minimum or maximum difficulty level, and optionally otherstatistical variants of the arrangement of the piece.

Step 26212 (“SHEET MUSIC PROCESSING (DIFFICULTY LEVELS)”) may forexample include: a music sheet of an arrangement of the selected piecemay be generated or retrieved in accordance with the appropriatedifficulty level and the user's playing level. If no assessment of theuser's level is available, a default level may be assumed. Thearrangement may be generated human generated or based on rules,crowdsourcing, AI techniques using engines trained upon labeledmulti-level music pieces, or the like. The music sheet may also beretrieved form a database comprising existing arrangements, or generatedby a combination of the above (e.g., retrieving automatically creatingan arrangement followed by human or crowd review or improvement), or thelike. When a new arrangement has been created, it may be stored forfuture users, thus enriching a library of arrangements of variousdifficulty levels. Step 26212 may thus provide an arrangement that fitsthe user's selections and level.

Step 26216 (“CREATE PRACTICE LEVELS”) may for example include: practicesessions may be generated based on the arrangement and difficulty level,comprising for example scraps of the arrangement, possibly of higher orlower difficulty levels. The practice sessions may be based on learningfrom previously created music sheets of the relevant levels, by offeringhigher difficulty sequences relevant to the arrangement, by learningwhere other users fail at that difficulty level and suggesting them aspractice, or the like. The practice sessions may be manually arranged bya musician and/or crowd sourcing. In some examples, the practicesessions may be automatically created using, for example, an artificialintelligence functionality. The practice sessions can also includeeducational videos, such as offline or live on-demand with humanteachers, rhythm training, learning of new notes or chords, or the like.The practice sessions may also include automatically generated teachingvideos.

Step 26220 (“Presentation & Understanding and/or Capture of Userplaying”) may for example include: the music sheet of the appropriatedifficulty level may be presented to the user, and the user's playingmay be captured. The presentation may take one or more forms, such asvisual user interface, voice instructions, augmented reality, or thelike.

The user may be playing alone or in a jam session together with others,with or without the BGM. For example, if the user plays the piano, theBGM may include drums, guitar, trumpet, bass, or vocals.

While the user is playing the displayed notes or chords, recognitionquality results and scores may be generated, presented online, andstored, based on processing the user playing, suppressing backgroundmusic and output recognition and progress results.

Step 26252 (“RECORD MICROPHONE(S)”) may for example include: the sound,including the user's playing, the played BGM, environmental noises andoptionally playing by other users, may be recorded by the selectedmicrophone(s), and provided to recognition and separation step 26240.

Step 26240 (“SEPARATION AND RECOGNITION”) may for example include: whilethe user plays, the sound information other than the user playing may beextracted from the received (e.g., captured and recorded) soundinformation, such that the component of the user's playing may berecognized, e.g., for the purpose of recognition and comparing with oneor more corresponding notations.

Non-limiting examples of notations may include any visual expression ofmusic being or to be played such as notes, chords, tabs, rhythmnotations, color indications, scores, illustrations, figures of merit(also: scores), text, and/or the like. The notations may be descriptiveof note pitch, length, note value, chords, key, tempo, instrumentation,and/or any other relevant music score information. In addition to therecognition results, recognition score may be determined, such as anumerical value or a verbal score, for example recognition OK, norecognition, error, unknown, probabilistic measure, compare to expectednote(s), chord(s) or other music expressions instructions, and ignore(for repeated error). In some embodiments, a probabilistic measure, or acomparison to expected note(s) may be output. Segments or sequences inwhich the recognition is poor may be collected over time and saved forfuture practice. The term segment or sequence may relate to any issuewith the user's performance, including a sequence of notes, tempo,rhythm, technique, notes, chords, transitions, or the like.

The separation and recognition results may be used on step 26228,wherein some parts of the displayed music sheet may be highlighted, forexample parts in which the user needs to improve, while other parts maybe left as is or just ignored. As the user practices and improves, thelabeling of the segments or sequences may be changed indicating theuser's progress, thus also providing the user with a sense ofimprovement and good experience.

Step 26232 (“Practice User Playing and Change Difficulty Level”) may forexample include: the user level estimation may be updated. For example,if the user plays the arrangement well, it may be assumed that thearrangement is easy, and vice versa. The difficulty level of the musicsheet presented to the user on step 26220 may then be changedaccordingly.

Step 26236 (“Panning microphone(s) and/or Speaker(s) selection”) may forexample include: the microphones and speakers to be used during thesession may be selected, in order to obtain a combination that yieldsthe best experience for the user including hearing well the backgroundmusic, while enabling the system to separate well the user's playingfrom the total recorded sound, thus providing for high qualityrecognition and monitoring of the user's playing. The I/O devices andtheir configurations are selected and may be altered online to maximizeperformance. The selection of the I/O devices may also influence the BGMpreparation. The I/O device selection and uses thereof are furtherdetailed in association with FIG. 27 below.

Step 26244 (“BGM Preparation”) may for example include: the BGM may beenhanced, for example in accordance with the difficulty level.Generally, a lower difficulty level implies a need for more concreteBGM, or in other words BGM that is closer to the tune, in order tocompensate for the basic level of the music as played by the user.

Step 26256 (“BGM Manipulation Volume, frequencies, Delete/move parts”)may for example include: the BGM may be manipulated by performingadditional signal processing on the BGM, for example changing thevolume, applying filters such as high or low pass filters, repeating ordeleting parts in accordance with the user's playing, removal offrequencies expected in the user-generated sound, or the like.

Step 26240 (“SEPARATION AND RECOGNITION”) may for example include:playing the BGM and monitoring the user's playing may receive the outputof BGM manipulation step 26256.

In some examples, steps 26232, 26244 and 26256 may be performedcontinuously or periodically based in real time or near real time on therecognition and the separation assessment in real-time.

Step 26248 (“Remove other (e.g., online) players and/or music devices)may for example include: when the user is playing in a jam session withother users, whether present in the same location or heard through aspeaker, the audio of the other users playing may also be suppressed orremoved from the captured audio, using the knowledge about what theother users are expected to be playing.

Step 26260 (“Play BGM (via Speakers)”) may for example include: the BGMas enhanced and updated on steps 26244 and 26256, respectively, may beplayed through the selected speakers.

Step 26224 (“Autogenerate Practice Level Based on User playing”) may forexample include: once the user has finished playing, the user may bepresented with the analysis score(s), and suggested to take apersonalized new practice session that is based on the user'sperformance in playing the presented music sheet, and the obstacles theuser faced while playing. If the user wishes to play the music sheetagain, a new music sheet may be created upon the playing performancewith the adequate difficulty level arrangement.

The following is an example pseudocode implementation of the diagramshown in FIG. 26 .

1. Initialization

-   -   Get device type    -   Get information on known default settings for that device    -   Load previous known settings for the user    -   If ((Device is the same as in setting) AND (Device orientation        the same) AND (Device playing scenario is the same)) then        -   Skip initialization    -   Else // Initializing        -   If (exist default setting for device)        -   Load default settings according to orientation        -   Else        -   Set random or initial guess based on other devices            configuration settings            2. Set Configuration based on initialization process            3. Predict configuration based on video, sound, static or            moving mode with no BGM    -   When sound is played—instructions/video—find a preferred        combination of microphone/speaker and/or settings for example,        by employing a search & testing method (e.g., round robin        manner)    -   Check the residuals after using BGM suppression algorithm    -   In static or moving mode without BGM—find the best microphone to        pick the sound and check that all relevant frequencies can be        heard    -   Integrate both checks—into a pair of microphone and speaker that        yields “best” expected separation for this configuration        (device, position, speaker, microphone, musical device, etc.)

Referring now to FIG. 27 showing a flowchart of steps in a method forselecting and configuring I/O devices to be used when teaching a user toplay, in accordance with some embodiments of the disclosure. The methodmay be used when the user is using the application or program forlearning to play a musical instrument, optionally with BGM and furtheroptionally with other users in a jam session.

Step 27304 (“OBTAIN INITIAL CONFIGURATION”) may for example include:initial configuration and settings may be obtained, obtaining an initialconfiguration including selection of at least one speaker and at leastone microphone of a computerized device used by the user. Step 304 mayfurther include preparing manipulation of the BGM, such as volume,frequency filtering, deletion or shifting of notes, or the like.

Step 27308 (“SET INITIAL CONFIGURATION”) may for example include: theinitial configuration and settings may be applied, e.g., by setting afirst parameter of the at least one speaker or the at least onemicrophone in accordance with the initial configuration, or setting themanipulation of the BGM as prepared on step 27304.

Step 27312 (“PROVIDE INSTRUCTIONS TO USER”) may for example include: anapplication used by the user may provide to the user music expressionsinstructions to be played, for example by displaying notes or chordsover a display device associated with the computing platform executingthe application.

Step 27316 (“PLAY BGM”) may for example include: the application mayfurther play the manipulated BGM of a piece selected by the user throughthe at least one speaker configured in accordance with the initialconfiguration.

Step 27320 (“CAPTURE SOUND”) may for example include: capturing soundincluding the user playing, and optionally the BGM, other users and/orambient noise may be captured through the at least one microphone.

Step 27324 (“PROCESS SOUND TO SUPPRESS BGM”) may for example include:the captured sound may be processed, using for example signal processingalgorithms, to suppress the BGM and optionally other users or backgroundnoise, to generate (e.g., filter out) a residual signal that is mainlyor solely descriptive of the user-generated sound. In other words theresidual signal pertains a core signal or signal of interest based onwhich, for example, a feedback is provided to the user, e.g., inreal-time when playing an instrument and/or when singing.

Step 27328 (“RECOGNIZE RESIDUAL SOUND”) may for example include: theresidual sound may be recognized to transcribe notes, chords or othermusic expressions instructions played by the user.

If or when a stopping criteria is met (step 27330), for example therecognition results, or other parameter such as the sound quality of thecaptured sound, the BGM, the user's sound, success of the user incompleting playing tasks or others is below a threshold, then thefollowing steps may be taken:

Step 27332 (“Determine Updated Configuration”) may for example include:an updated configuration and setting comprising an updated selection ofthe at least one speaker and at least one microphone may be determined,in accordance with the recognition results, and the sound quality; andmanipulation of the BGM. such as updating the volume, filteringfrequencies, deletion or shifting of notes, etc.

Step 27336 (“SET UPDATED CONFIGURATION”) may for example include: theupdated configuration and settings may be applied, by setting a secondparameter of the at least one speaker or the at least one microphone inaccordance with the updated configuration and manipulation of the BGM,thereby improving the recognition results. Applying the updatedconfiguration and setting may also improve the sound quality of thecaptured sound, the BGM, the user's sound or the like. In someembodiments, determining and updating the configuration may be performedwithout checking the stopping criteria, regardless of whether it is metor not.

Referring now to FIG. 28 , showing a detailed flowchart of steps in amethod for selecting and configuring I/O devices to be used whenteaching a user to play, in accordance with some embodiments of thedisclosure.

Step 28404 (“NEW PLAY”) may for example include: the user may initiate anew playing session, and indicate a piece he wishes to play.

Step 28408 (“Expected playing (e.g., sheet music)”) may for exampleinclude: a music sheet of an arrangement to be played by the user may beobtained, for example retrieved from a database, composed by anartificial intelligence engine and optionally enhanced by a human or bycrowdsourcing, or the like.

Step 28412 (“BGM”) may for example include: the BGM to be played withthe arrangements may be obtained similar to the music sheet, comprisingfor example music by drums or other instruments, singing, or the like.

Step 28416 (“Jamming other users expected playing”) may for exampleinclude: music sheets to be played by other users participating in a jamsession with the user may be obtained as above.

Steps 28420 (“Past settings/user preferences”), 28424 (“Default/usermic(s)/speaker(s) selection”) and 28428 (“Obtain Device I/O output”) mayfor example include: information may be obtained, which may contain datarelevant for selecting and configuring the I/O devices. The informationmay be obtained from a storage device associated with the computingplatform used by the user, from a remote database accessible to thecomputing platform, provided by the user, or the like.

Step 28420 (“Past settings/user preferences”) may for example include:information regarding the user's past performance and preferences inplaying notes or chords, sequences, or other techniques. The informationmay be collected during past sessions in which the user played.

Step 28424 (“Default/user mic(s)/speaker(s) selection”) may for exampleinclude: information regarding past or default settings and selectionsrelated to the I/O devices selected by or for the user may be obtained.

Step 28428 (“Obtain Device I/O output”) may for example include: inputmay be received from sensors associated with the device, such ascameras, gyros, or others, which may provide information useful fordetermining the device current situation, such as position and/ororientation, whether an I/O device is blocked, what is the environmentof the user (for example indoor/outdoor), whether the device is stablylocated or not, how noisy the environment is, or the like.

In some examples, information gathered on one or more of the abovesteps, e.g., 28408, 28412, 28416, 28420, 28424 or 28428 may be gatheredand used on initial configuration and settings obtaining step 28432(“Obtain initial configuration”). Obtaining may relate to computing theconfiguration by taking into account past configurations, and updatingthem in accordance with the music sheet and the additional audio, suchas the BGM and the other users, and further enhancing in accordance withdata obtained from additional sensors which may provide information onthe status of the device, the user, and availability of the I/O devices.

In some embodiments, obtaining may relate to retrieving theconfiguration from a storage device, receiving the configuration over acommunication channel, or the like.

It will be appreciated that additional factors may also be considered indetermining the initial configuration and settings, such as feasible andinfeasible I/O device combinations.

Step 28436 (“Set configuration”) may for example include: the initialconfiguration and settings may be applied, e.g., the relevant I/Odevices may be set with the selected settings, comprising setting atleast one parameter of a selected speaker or a selected microphone inaccordance with the obtained configuration.

Step 28440 (“Activate I/O devices”) may for example include: one or moreI/O devices may be activated, for example, in accordance with theinitial settings and configurations applied in step 28436.

Step 28444 (“BGM Manipulation”) may for example include: the BGM may bemanipulated in accordance with the I/O device selection andconfiguration. For example, a speaker playing the BGM may be set withthe correct volume level, certain frequencies may be compressed, the BGMmay be shifted in time to comply with the user's pace, or the like.Since, as detailed below the sound is to be separated and the user'splaying is to be recognized, it will be appreciated that if two or moremanipulation options are expected to provide the same or similarseparation result, for example a result that differs from another resultin no more than a predetermined threshold, the manipulation option thatintroduces fewer and smaller changes to the BGM may be preferred.

Step 28448 (“BGM played, notes displayed, sound captured”) may forexample include: optionally equivalent to step 26220 of FIG. 26 , theapplication may display to the user notes to be played over the displaydevice associated with the computing platform executing the application.The application may further play the BGM as manipulated and optionallyplay music by other users, and capture the sound comprising the user'splaying, the BGM and the other user's playing.

Step 28452 (“Process sound to suppress BGM, Jamming and/or Noise”) mayfor example include: the sound captured on step 28448 may be processed,using signal processing to suppress the BGM and the sound by otherjamming users, to obtain residual sound. The suppression may beperformed by multiple codec algorithms, such as Speex available fromhttps://www.speex.org/, Opus available from https://opus-codec.org/orwebrtc available from https://webrtc.org/.

Step 28456 (“Recognize residual sound”) may for example include: theresidual sound may be recognized, i.e., the notes or chords played bythe user may be transcribed. In some embodiments, recognition may beperformed on the sound as captured, as well as on the residual sound,and the results may be combined, compared, or the like. The recognitionquality, which may be measured according to one or more aspects, e.g.,the number or percentage of notes that were successfully recognized, maybe evaluated.

Step 28460 (“Record Sound”) may for example include: simultaneously withsteps 28452 and 28456, the audio may be recorded.

Step 28464 (“Analyze Sound and recognition results”) may for exampleinclude: the recognition quality or results, as well as the recordedaudio being the combination of the BGM, the residual sound andoptionally other users and background noise may be analyzed. Lowrecognition quality means that the system was unable to recognizecorrectly a sufficient number or percentage of the notes or chordsplayed by the user, and will thus provide poor feedback to the userregarding a level of correspondence between the displayed musicalnotations and the user-generated sound regarding, for example,correctness of the played notes, the rhythm, the intensity, and/or thelike. Therefore, it is preferred that the system is configured such thatthe BGM and other users' playing suppression can be performed in asatisfactory manner to enable high or sufficient quality recognition,for example by having a volume, pitch or another difference exceeding apredetermined threshold. In some embodiments, the feedback may beprovided in real-time.

The expression “real-time” as used herein generally refers to theupdating of information at essentially the same rate as the data isreceived. For example, in the context of the present disclosure,“real-time” is intended to mean that the sound is captured, processedfor providing, for example, a feedback at a short enough time delay suchthat the feedback is provided without user-noticeable judder, latencyand/or lag.

Thus, if at least one aspect of the recognition quality is below apredetermined threshold, for example the system was unable to recognizeat least a predetermined number or percentage of the notes played by theuser, changes may be introduced to the BGM and to the configuration onsteps 28468 and 28472, respectively, as detailed below.

Analyzing the sound may also include determining feedback to be providedto the user, for example whether and where the user has erred, what arethe user's errors for example wrong notes, rhythm, or other problems, orthe like. The feedback may then be provided to the user, for exampledisplayed over the display device of the computing platform used by theuser.

Step 28468 (“Update BGM”) may for example include: the BGM may beupdated in accordance with the analysis results, for example somefrequencies may be increased or silenced if the recognition rate islower than a threshold, or the BGM may be made closer to the tune if theuser's error rate or error number exceeded a threshold, or vice versa.Additionally, recommendations may be determined, such as telling theuser to play louder, prompt the user that the environment is too noisy,or the like.

Step 28472 (“Update configuration”) may for example include: the I/Odevices selection and settings may be updated in accordance with theanalysis results and with the current status of the I/O devices. In someembodiments, a configuration from a plurality of candidateconfigurations of one or more microphones and/or settings, and,optionally, where applicable, one or more speakers and/or settings, maybe obtained (e.g., selected), for example by employing a search and/ortesting method such as, for instance, an exhaustive search (e.g., roundrobin algorithm), a heuristic method and/or the like. The selectedcombination may herein be referred to as a “preferred” combination, forproviding an improved or optimized user experience.

The residual sound obtained on step 28452 may then be evaluated per eachsuch combination, and the combination may be assigned a score, dependingon the quality of the sound separation. The separation quality may beassessed in accordance with the SNR for each combination, therecognition rate, success of the user in completing playing tasks, orthe like.

In situations where the user is playing without BGM and/or without otherusers, and/or in a static manner, in which the user is only displayedwith the notes and plays at his own pace, it may be determined which ofthe one or more microphones and/or microphone settings (e.g., frequencyprocessing settings) provides comparatively improved or optimized userexperience. The user experience may be assessed, for example, inaccordance with the SNR for each combination, a recognition rate,success of the user in completing playing tasks, and/or the like.

In some embodiments, if one or more speakers and/or microphones becomesunavailable and/or is covered due to change in the position of thedevice, and/or if the BGM frequencies are better adapted to anotherspeaker, then the used I/O devices or their parameters may be changed,comprising selecting another I/O device or setting at least oneparameter of a speaker and/or a microphone to a different value.

The updated configuration may also be aimed at improving the userexperience, e.g., providing better balance in intensity between the BGM,other users and the user's playing.

Step 28476 (“Store Parameters”) may for example include: the selecteddevices and settings may be stored. The selections and settings may beretrieved on step 28420 in a future session.

The following is an example pseudocode implementation of the diagramshown in FIG. 28 . Start using the application—playing with BGM

-   -   Application start—BGM playing and presenting the part to be        played    -   Microphone and speaker are open based on decision    -   Manipulation to the BGM to yield better results (if needed)    -   Compression of frequencies    -   Change volume    -   Shift the BGM in time    -   Remove/suppress the BGM from the recoding    -   Perform recognition based on the input (recording with/without        BGM suppression and expected notes (if any)    -   In parallel record the original microphone input    -   Analyze the recognition results, given the settings, sound        quality and the BGM. If recognition is insufficient perform one        or more of the following:    -   Change microphone/speaker selection—round robin or another        method    -   Change the manipulation of the BGM    -   Set a new configuration if needed—input back into the setting        module, start the process over.        If (End of piece)    -   End of BGM playing    -   Save the final parameters for next session

Additional reference is now to FIG. 29 , showing a block diagram of asystem for selecting I/O devices when teaching a user to play, inaccordance with some embodiments of the disclosure.

FIG. 29 shows components related to selection and setting of the I/Odevices. It will be appreciated that other components may be included inorder to provide a full system for teaching a user to play a musicalinstrument in accordance with the disclosure.

The apparatus may comprise computing device 29500 such as tablet orsmartphone 25108 of FIG. 25 . Computing device 29500 may comprise one ormore processors 29504. Any of processors 29504 may be a CentralProcessing Unit (CPU), a microprocessor, an electronic circuit, anIntegrated Circuit (IC) or the like. Processors 29504 may be utilized toperform computations required by computing device 29500 or any of itsubcomponents, for example steps in the methods of FIG. 26 , FIG. 27 orFIG. 28 above.

Computing device 29500 may comprise one or more speakers 29508. In someembodiments, one or more of speakers 29508 may be external to computingdevice 29500.

Computing device 29500 may comprise one or more microphones 29512. Insome embodiments, one or more of microphones 29512 may be external tocomputing device 29500.

Computing device 29500 may comprise communication module 516, forcommunicating with a server such as server 25128 of FIG. 25 , withdatabases storing music pieces and arrangements, or the like.

Computing device 29500 may comprise additional I/O devices and/orsensors 29518 including, for example, inertial and/or non-inertialsensors such as cameras, linear acceleration sensors, angularacceleration sensors, gyroscopes, satellite-based navigation systems(e.g., the US-based Global Positioning System). Microphones 29512 and/oradditional I/O devices and/or sensors 29518 may be may employed, forexample, to identify the position of the device, the distance from theuser and from the musical instrument, the type of music instrument beingplayed by the player or players, and/or the like. In some embodiments,recognition may be performed by fusing visual and audio sources. Forexample, a back camera can identify that the back cover covers amicrophone.

Computing device 29500 may comprise one or more storage devices 29520for storing data structures and/or program code, executable by processor29504 to result in the implementation and/or execution of one or moremodules and/or processes. For example, storage device 29520 may retaindata structures and program code which, when executed by any ofprocessors 504 and/or processors 29504 to perform acts associated withany of the methods, processes, procedures and/or steps described herein,e.g., as described with respect to FIG. 26 , FIG. 27 or FIG. 28 .

Storage device 29520 may be persistent or volatile. For example, storagedevice 29520 can be a Flash disk, a Random Access Memory (RAM), a memorychip, an optical storage device such as a CD, a DVD, or a laser disk; amagnetic storage device such as a tape, a hard disk, storage areanetwork (SAN), a network attached storage (NAS), or others; asemiconductor storage device such as Flash device, memory stick, or thelike.

The components detailed below may be implemented as one or more sets ofinterrelated computer instructions, executed for example by any ofprocessors 29504 and/or by another processor. In some embodiments, someof the components may be executed by computing device 29500 while othersmay be executed by another computing platform such as server 25128. Thecomponents may be arranged as one or more executable files, dynamiclibraries, static libraries, methods, functions, services, or the like,programmed in any programming language and under any computingenvironment.

In some exemplary embodiments of the disclosed subject matter, computingdevice 29500 (e.g., storage device 29520) may accommodate one or moredrivers 29524, for receiving data from any devices, and in particularany of speakers 29508 and microphones 29512. Drivers 29524 may also beoperative in setting the operational parameters of the devices sdetermined, for improving the user experience and enabling recognitionof the sound.

Computing device 29500 (e.g., storage device 29520) may accommodate dataobtaining module 29528 for obtaining past user selections andpreferences, past selected I/O devices and relevant settings, or thelike. The data may be obtained from a database, for example a databasebeing stored on storage device 29520 or another storage deviceoperatively connected to computing device 29500. Data obtaining module29528 may be further operative in obtaining available musical pieces,arrangements, BGM or the like.

Computing device 29500 (e.g., storage device 29520) may accommodatearrangement, BGM and jam users' roles determination module 29532, fordetermining the roles for the user, for additional users if any, andcorresponding BGM music to be played, in accordance with a selectedpiece, and difficulty level.

Computing device 29500 (e.g., storage device 29520) may accommodateconfiguration determination module 29536, for determining an I/Oconfiguration, including selecting speaker(s) through which the BGMmusic and playing by additional users (or expected playing) is to beplayed, microphone(s) through which the user's playing, the BGM and theadditional users' playing are to be captured, and the relevant settingsthereof.

Computing device 29500 (e.g., storage device 29520) may accommodatesound separation module 29540 for suppressing from the captured audiothe BGM, the other users' playing and environmental noises. Suppressingmay use the available knowledge about the BGM and of the other user'splaying, or at least their expected sound.

Computing device 29500 (e.g., storage device 29520) may accommodaterecognition module 29544 for transcribing the audio comprising only ormainly the user's playing, and obtaining the notes or chords played bythe user.

Computing device 29500 (e.g., storage device 29520) may accommodateanalysis module 29548, for comparing the transcribed notes or chords andtheir timing to the expected notes or chords according to the musicsheet provided to the user, identifying errors, determining if adifficulty level change is required, or the like.

Computing device 29500 (e.g., storage device 29520) may accommodate userinterface 29552, for displaying a music sheet to a user over a displaydevice, displaying a cursor tracking the user's playing or what the useris expected to be playing, displaying results such as error report tothe user, receiving user's preferences, or the like.

Computing device 29500 (e.g., storage device 29520) may accommodatecontrol and data flow module 29556 for invoking the relevant modules,providing the required data or access to the data to each module,collecting results, or the like.

Computing device 29500 (e.g., storage device 29520) may also storeexecution results, for example the selected configuration and settingsper user, per device orientation and position, per arrangement, perdifficulty level, or the like. Once stored, the configuration andsettings may be retrieved and used as an initial configuration andsettings for another session by the same user or by other users, forexample users with similar preferences and level.

It will be appreciated that processor 29504 and/or a processor of server25128 is operable to execute methods, processes and/or operationsdescribed herein. For instance, processor 29504 may execute program codeinstructions resulting in the implementation of, for example, modules29524, 29528, 29532, 29536, 29540, 29544, 29548, 29552 and 29556 and/orresult in the implementation of a method for providing a music learningsession, including, for example, selecting and setting I/O devices forimproving the recognition quality and enhancing the user experience.

ADDITIONAL EXAMPLES

Example 1 pertains to a method for providing a session of music learningto a user, comprising

setting a sound capturing configuration relating to at least onemicrophone of a computerized device;

providing the user with musical notations to be executed through theuser for generating user-generated sound;

capturing, through the at least one microphone, sound which includesuser-generated sound produced by the user to produce user-generatedsound data;

processing the user-generated sound data; and

determining, based on the processing of the user-generated sound data,whether the sound capturing configuration is to be adapted or not.

In Example 2, the subject matter of Example 1 may optionally furtherinclude adapting the sound capturing configuration based on theuser-generated sound data.

In Example 3, the subject matter of Examples 1 or 2 may optionallyfurther include, wherein a sound capturing configuration pertains to aconfiguration of at least one microphone.

In example 4, the subject matter of any one or more of the precedingexamples may optionally include wherein a sound capturing configurationpertains to a selection of one or more microphones of a plurality ofmicrophones.

In example 5, the subject matter of any one or more of the precedingexamples may optionally include, wherein adapting the sound capturingconfiguration includes adapting a configuration of the at least onemicrophone.

In example 6, the subject matter of any one or more of the examples 2 to4 may optionally include wherein adapting the sound capturingconfiguration includes adapting a microphone selection.

In example 7, the subject matter of any one or more of the precedingexamples may optionally include, wherein the captured sound furtherincludes an audio playback.

In example 8, the subject matter of example 7 may optionally includecomprising recognizing the user-generated sound from the captured sound.

In example 9, the subject matter of example 8 may optionally compriseadapting the sound capturing configuration to improve the recognizing ofthe user-generated sound from the captured sound also comprising theaudio playback.

In example 10, the subject matter of example 9 may further comprisedetermining, based on the user-generated sound data, whether an audioplayback configuration is to be adapted to improve recognizing of theuser-generated sound.

In example 11, the subject matter of example 10 may further compriseadapting, based on the user-generated sound data, the audio playbackconfiguration to improve recognizing of the user-generated sound.

In example 12, the subject matter of examples 10 and/or 11 mayoptionally include, wherein the audio playback configuration pertains toa speaker configuration.

In example 13, the subject matter of example 12 may optionally includewherein the speaker configuration pertains to a speaker outputconfiguration and/or to a speaker selection.

In example 14, the subject matter of any one or more of the examples 1to 13 may optionally comprise transcribing the user-generated sound intocorresponding notation.

In example 15, the subject matter of example 14 may optionally includewherein the notation is descriptive of one or more of the following: anote pitch, note value, tempo, meter, and key.

Example 16 pertains to a method for providing a session of musiclearning to a user, comprising:

obtaining a sound capturing and playback configuration;

setting a first parameter of the at least one speaker or the at leastone microphone in accordance with the initial configuration;

displaying to the user musical notations based on which the userproduces user-generated sound;

playing background music (BGM) of a piece selected by the user throughthe at least one speaker configured in accordance with the initialconfiguration;

capturing, through the at least one microphone, composite soundincluding the BGM and user-generated sound;

processing the captured composite sound to recognize the user-generatedsound; and

determining, based on the processing, whether the sound capturing and/orplayback configuration is to be adapted to improve recognizing theuser-generated sound.

In Example 17, the subject matter of example 16 may further compriseadapting, based on the recognized user-generated sound, the soundcapturing and/or playback configuration to improve recognizing theuser-generated sound.

In example 18, the subject matter of example 17 may further comprise,optionally, wherein the adapting of the sound capturing and/or playbackconfiguration includes setting a microphone and/or audio playbackconfiguration, and/or selecting at least one speaker of a plurality ofspeakers and/or selecting at least one microphone of a plurality ofmicrophones.

In example 19, the subject matter of examples 17 and/or 18 mayoptionally further comprise wherein the processing includes suppressingthe BGM.

In example 20, the subject matter of any one or more of the examples 16to 20 may optionally further comprise seizing displaying to the usermusical notations if a stopping criterion is met.

In example 21, the subject matter of example 20 may optionally furthercomprise wherein the stopping criteria pertains to recognition resultsbeing below a predetermined passing threshold in at least one aspectand/or failure of the user in completing playing tasks.

In example 22, the subject matter of any one or more of the examples 16to 21 may further comprise wherein the musical notations are descriptiveof one of the following: note pitch, length, chords, note value, key,tempo, instrumentation, or any combination of the aforesaid.

In example 23, the subject matter of any one or more of the examples 16to 22 may further comprise recognizing the user-generated soundcomprises determining a sound quality of the captured composite soundcomprising the BGM and the user-generated sound.

In example 24, the subject matter of example 23 may optionally furthercomprise adapting the sound capturing and/or playback configuration toimprove the sound quality of the captured composite sound.

In example 25, the subject matter of example 24 may optionally furthercomprise wherein an adapted sound capturing and/or playbackconfiguration provides an improved user experience compared to a userexperience prior to the adaptation.

In example 26, the subject matter of example 25 may optionally furthercomprise wherein the user experience is determined in accordance with asuccess in completing instrument playing and/or singing tasks inaccordance with the displayed musical notations.

In example 27, the subject matter of examples 25 or 26 may optionallyfurther comprise determining a level of correspondence betweenuser-generated sound and the displayed musical notes.

In example 28, the subject matter of example 27 may optionally furthercomprise wherein the user experience is determined based on the level ofcorrespondence.

In example 29 the subject matter of examples 27 or 28 may optionallyfurther comprise updating the BGM in accordance with the level ofcorrespondence.

In example 30, the subject matter of any one or more of the examples 16to 29 may optionally further comprise wherein the processing of thecaptured composite sound includes:

determining a feedback relating to the user-generated sound; and

providing the feedback to the user.

In example 31, the subject matter of any one or more of the examples 16to 30 may further comprise capturing through the at least onemicrophone, user-generated sound produced by at least one other user ina collaborative playing setting; and

providing, through the at least one speaker, an audio output relating tothe user-generated sound produced by the at least one other user suchthat the composite sound additionally includes user-generated soundproduced by the at least one other users.

In example 32, the subject matter of example 31 may optionally furthercomprise, wherein the processing of the composite sound is performed torecognize the user-generated sound of a selected user.

In example 33 the subject matter of example 32 may optionally furthercomprise wherein the processing comprises suppressing the role of the atleast one other user in the captured composite sound to recognize theuser-generated sound of the selected user.

In example 34 the subject matter of any one or more of the examples 16to 33 may optionally further comprise wherein the adapting of the soundcapturing and/or playback configuration is performed based on an outputreceived from a sensor including one or more of the following: a camera,a linear acceleration sensor, an angular acceleration sensor, agyroscope, and a satellite-based navigation system.

Example 35 pertains to a system configured to provide a music learningsession, comprising;

a memory for storing data and executable instructions; and

a processor that is configured to execute the execution instructions toresult in the following:

setting a sound capturing configuration relating to at least onemicrophone of a computerized device;

providing the user with musical notations to be executed through theuser for generating user-generated sound;

capturing, through the at least one microphone, sound which includesuser-generated sound produced by the user to produce user-generatedsound data;

processing the user-generated sound data; and

determining, based on the processing of the user-generated sound data,whether the sound capturing configuration is to be adapted or not.

In example 36, the subject matter of example 35 may optionally furthercomprise adapting the sound capturing configuration based on theuser-generated sound data.

In example 37, examples 35 or 36 may optionally further comprise whereina sound capturing configuration pertains to a configuration of at leastone microphone.

In example 38, any one or more of the examples 35 to 37 may optionallyfurther comprise, wherein a sound capturing configuration pertains to aselection of one or more microphones of a plurality of microphones.

In example 39, the subject matter of any or more of the examples 35 to38 may optionally further comprise wherein adapting the sound capturingconfiguration includes adapting a configuration of the at least onemicrophone.

In example 40, the subject matter of any one or more of the examples 35to 39 may optionally further comprise adapting the sound capturingconfiguration includes adapting a microphone selection.

In example 41, the subject matter of any one or more of the examples 35to 40 may optionally further comprise wherein the captured sound furtherincludes an audio playback.

In example 42, the subject matter of example 41 may optionally furthercomprise recognizing the user-generated sound from the captured sound.

In example 43, the subject matter of example 42 may optionally furthercomprise adapting the sound capturing configuration to improve therecognizing of the user-generated sound from the captured sound alsocomprising the audio playback.

In example 44 the subject matter of any one or more of the examples 35to 43 may optionally further comprise determining, based on theuser-generated sound data, whether an audio playback configuration is tobe adapted to improve recognizing of the user-generated sound.

In example 45, the subject matter of example 44 may optionally furthercomprise adapting, based on the user-generated sound data, the audioplayback configuration to improve recognizing of the user-generatedsound.

In example 46, the subject matter of examples 44 and/or 45 mayoptionally further comprise wherein the audio playback configurationpertains to a speaker configuration.

In example 47 the subject matter of example 46 may optionally furthercomprise wherein the speaker configuration pertains to a speaker outputconfiguration and/or to a speaker selection.

In example 48 the subject matter of any one or more of the examples 35to 47 may optionally further comprise transcribing the user-generatedsound into corresponding notation.

In example 49 the subject matter of example 48 may optionally furthercomprise wherein the notation is descriptive of one or more of thefollowing: a note pitch, note value, tempo, meter, and key.

Example 50 pertains to a system configured to provide a session of musiclearning to a user, comprising:

a memory for storing data and executable instructions; and

a processor that is configured to execute the execution instructions toresult in the following:

obtaining a sound capturing and playback configuration;

setting a first parameter of the at least one speaker or the at leastone microphone in accordance with the initial configuration;

displaying to the user musical notations based on which the userproduces user-generated sound;

playing background music (BGM) of a piece selected by the user throughthe at least one speaker configured in accordance with the initialconfiguration;

capturing, through the at least one microphone, composite soundincluding the BGM and user-generated sound;

processing the captured composite sound to recognize the user-generatedsound; and

determining, based on the processing, whether the sound capturing and/orplayback configuration is to be adapted to improve recognizing theuser-generated sound.

In example 51, the subject matter of example 50 may further compriseadapting, based on the recognized user-generated sound, the soundcapturing and/or playback configuration to improve recognizing theuser-generated sound.

In example 52 the subject matter of example 51 may further comprisewherein the adapting of the sound capturing and/or playbackconfiguration includes setting a microphone and/or audio playbackconfiguration, and/or selecting at least one speaker of a plurality ofspeakers and/or selecting at least one microphone of a plurality ofmicrophones.

In example 53 the subject matter of any one or more of the examples 50to 52 may further comprise wherein the processing includes suppressingthe BGM.

In example 54 the subject matter of any one or more of the examples 50to 53 may further comprise, further comprising seizing displaying to theuser musical notations when a stopping criterion is met.

In example 55 the subject matter of example 54 may optionally furthercomprise wherein the stopping criteria pertains to recognition resultsbeing below a predetermined passing threshold in at least one aspectand/or failure of the user in completing playing tasks.

In example 56 the subject matter of any one or more of the examples 50to 55 may optionally further comprise wherein the musical notations aredescriptive of one of the following: note pitch, length, chords, notevalue, key, tempo, instrumentation, or any combination of the aforesaid

In example 57 the subject matter of any one or more of the examples 50to 56 may optionally further comprise wherein recognizing theuser-generated sound comprises determining a sound quality of thecaptured composite sound comprising the BGM and the user-generatedsound.

In example 58 the subject matter of example 57 may optionally furthercomprise: adapting the sound capturing and/or playback configuration toimprove the sound quality of the captured composite sound.

In example 59 the subject matter of example 58 may optionally furthercomprise wherein an adapted sound capturing and/or playbackconfiguration provides an improved user experience compared to a userexperience prior to the adaptation.

In example 60 the subject matter of example 59 may optionally furthercomprise wherein the user experience is determined in accordance with asuccess in completing instrument playing and/or singing tasks inaccordance with the displayed musical notations.

In example 61 the subject matter any one or more of the examples 50 to60 may optionally further comprise: determining a level ofcorrespondence between user-generated sound and the displayed musicalnotations

In example 62, the subject matter of example 61 may optionally furthercomprise wherein the user experience is determined based on the level ofcorrespondence.

In example 63, the subject matter of examples 61 and/or 62 mayoptionally further comprise updating the BGM in accordance with thelevel of correspondence.

In example 64, the subject matter of any one or more of the examples 50to 63 may optionally further comprise wherein the processing of thecaptured composite sound includes:

determining a feedback relating to the user-generated sound; and

providing the feedback to the user.

In example 65, the subject matter of any one or more of the examples 50to 64 may optionally further comprise:

capturing through the at least one microphone, user-generated soundproduced by at least one other user in a collaborative playing setting;and

providing, through the at least one speaker, an audio output relating tothe user-generated sound produced by the at least one other user suchthat the composite sound additionally includes user-generated soundproduced by the at least one other users.

In example 66 the subject matter of example 65 may optionally furthercomprise, wherein the processing of the composite sound is performed torecognize the user-generated sound of a selected user.

In example 67 the subject matter of example 66 may optionally furthercomprise wherein the processing comprises suppressing the role of the atleast one other user in the captured composite sound to recognize theuser-generated sound of the selected user.

In example 68 the subject matter of any one or more of the examples 50to 67 may optionally further comprise, wherein the adapting of the soundcapturing and/or playback configuration is performed based on an outputreceived from a sensor including one or more of the following: a camera,a linear acceleration sensor, an angular acceleration sensor, agyroscope, and a satellite-based navigation system.

Example 69 pertains to a method for use with a musical piece, or partthereof, comprising: presenting, by at least one client device to aplurality of users, a corresponding plurality of sequences of musicalsymbols to be cooperatively played by the plurality of users using aplurality of instruments;

capturing instrument outputs produced by the plurality of instruments togenerate audio data descriptive of the captured instrument outputs;analyzing the audio data for determining a level of correspondencebetween the captured instrument outputs and the plurality of musicalsymbols presented to the plurality of users; andpresenting, based on the level of correspondence, an adapted ornon-adapted sequence of music symbols at least one of the plurality ofplayers.

In Example 70 the subject matter of Example 69 may optionally furthercomprise wherein the at least one client device outputs accompanyingbackground music (BGM) that corresponds with the plurality of musicalsymbols presented to the plurality of users.

In example 71, the subject matter of examples 69 and/or 70 mayoptionally further comprise wherein the cooperative playing ofinstruments by the plurality of users includes playing the instrumentsin timed coordination, e.g., at a same location or at differentlocations.

In Example 72, the subject matter of any one or more of examples 69 to71 may optionally further comprise wherein the cooperative playing ofinstruments by the plurality of users includes playing the instruments(e.g., in different time periods or a same time period for instance intimed coordination or substantially timed coordination), e.g., at a samelocation or at different locations.

In Example 73, the subject matter of any one or more of examples 69 to72 may optionally further comprise capturing the instrument outputs(e.g., in different time periods or a same time period for instance intimed coordination or substantially timed coordination), e.g., at a samelocation or at different locations; and merging the captured instrumentoutputs to arrive at a merged instrument output.

In example 74, the subject matter of example 73 may optionally furthercomprise analyzing a level of correspondence of the merged instrumentoutput with the plurality of sequences of musical symbols presented tothe users.

In example 75, the subject matter of any one or more examples 69 to 74may optionally further comprise wherein the instrument outputs comprisesounds captured by a microphone and/or audio signals captured by a wiredor wireless receiver device.

Example 76 pertains to a system for use with a musical piece, or partthereof, comprising: one or more processors; and one or more memoriesstoring software code portions executable by the one or more processorsto cause the system to perform the following:

presenting, by at least one client device to a plurality of users, acorresponding plurality of sequences of musical symbols to becooperatively played by the plurality of users using a plurality ofinstruments;capturing instrument outputs produced by the plurality of instruments togenerate audio data descriptive of the captured instrument outputs;analyzing the audio data for determining a level of correspondencebetween the captured instrument outputs and the plurality of musicalsymbols presented to the plurality of users; andpresenting, based on the level of correspondence, an adapted ornon-adapted sequence of music symbols at least one of the plurality ofplayers

In Example 77, the subject matter of example 76 may optionally furthercomprise wherein the at least one client device outputs accompanyingbackground music (BGM) that corresponds with the plurality of musicalsymbols presented to the plurality of users.

In Example 78, the subject matter of examples 76 and/or 77 mayoptionally further comprise wherein the cooperative playing ofinstruments by the plurality of users includes playing the instrumentsin timed coordination (e.g., synchronously) or in substantially timedcoordination, e.g., at a same location or at different locations.

In example 79, the subject matter of any one or more of examples 76 to77 may optionally further comprise wherein the cooperative playing ofinstruments by the plurality of users includes playing the instrumentsin different time periods, e.g., at a same location or at differentlocations.

In example 80, the subject matter of any one or more of the examples 76to 79 may optionally further comprise capturing the instrument outputsresponsively produced by the plurality of users; and merging thecaptured instrument outputs to arrive at a merged instrument output.

In example 81, the subject matter of example 80 may optionally furthercomprise analyzing a level of correspondence of the merged sound withthe plurality of sequences of musical symbols presented to the users.

In example 82 the subject matter of any one or more of the examples 76to 81 may optionally further comprise wherein the instrument outputscomprise sounds captured by a microphone and/or audio signals capturedby a wired or wireless receiver device.

Example 83 pertains to a method for use with a musical piece, or partthereof, that is cooperatively played by a first musical instrumentaccording to first sequence of musical symbols, by a second musicalinstrument according to second sequence of musical symbols, and by athird musical instrument according to third sequence of musical symbols,for use with a first client device associated with the first musicalinstrument and operated by a first person that is identified by a firstperson identifier, with a second client device associated with thesecond musical instrument and operated by a second person that isidentified by a second person identifier, and with a third client deviceassociated with the third musical instrument and operated by a thirdperson that is identified by a third person identifier, wherein each ofthe first, second, and third client devices comprises one of thefollowing: a microphone, a digital signal connection, a sounder, and apresentation or display and communicates over the Internet and/or anyother communication network with a server device that stores the first,second, and third sequences, the method comprising:

(e.g., synchronously) sending, by the server device to the first,second, and third client devices, respectively the first, second, andthird sequences of musical symbols;receiving, by first, second, and third client devices from the serverdevice, respectively the first, second, and third sequences;displaying, to the first, second, and third persons by the respectivedisplay in the respective first, second, and third client devices, therespective received first, second, and third sequences;capturing, by the respective microphone in the first, second, and thirdclient devices, respective first, second, and third sounds from therespective first, second, and third musical instruments;sending, by the first, second, and third client devices to the serverdevice, the respective captured first, second, and third sounds;receiving, by the server device from the first, second, and third clientdevices, the respective captured first, second, and third sounds;analyzing, by the server device, the received captured first, second,and third sounds;sending, by the server device only to the second and third clientdevices, the received captured first sound;receiving, by the second and third client devices from the serverdevice, the received captured first sound;emitting, by the sounder in the second and third client devices, thereceived captured first sound;sending, by the server device only to the third client device, thereceived captured second sound;receiving, by the third client device from the server device, thereceived captured second sound; andemitting, by the sounder in the third client devices, the receivedcaptured second sound.

In example 84, the subject matter of 83 may optionally further comprisesending, by the server device to the first client device, the receivedcaptured second and third sounds;

receiving, by the first client devices from the server device, thereceived captured second and third sounds; andemitting, by the sounder in the first client device, the receivedcaptured second and third sounds.

In example 85, the subject matter of examples 83 and/or 84 mayoptionally further comprise sending, by the server device to the secondclient device, the received captured third sound;

receiving, by the second client device from the server device, thereceived captured third sound; andemitting, by the sounder in the second client device, the receivedcaptured third sound.

In example 86, the subject matter of any one or more examples 83 to 85may optionally further comprise checking, by the server device, whetherthe each of the received captured sounds matches the respectivesequence; determining, by the server device, the amount of musicalsymbols in each of the sequences that do not match with the respectivecaptured sound; and updating, by the server device, the skill levelvalue associated with the respective person identifier in response tothe amount of the musical symbols that do not match in the respectivecaptured sound.

In example 87, the subject matter of any one or more of examples 83 to86 may optionally further comprise, for use with a threshold, andwherein the updating comprises raising the skill level value associatedwith the person identifier in response to the amount of the non-matchingmusical symbols being less than the threshold.

In example 88, the subject matter of any one or more of examples 83 to87 may optionally further comprise for use with a threshold, and whereinthe updating comprises lowering the skill level value associated withthe person identifier in response to the amount of the non-matchingmusical symbols being more than the threshold.

Example 89 pertains to a method for use with a musical piece, or partthereof, comprising: presenting, by at least one client device to aplurality of users, a corresponding plurality of sequences of musicalsymbols to be cooperatively played by the plurality of users using aplurality of instruments; capturing instrument outputs produced by theplurality of instruments to generate audio data descriptive of thecaptured instrument outputs; analyzing the audio data for determining alevel of correspondence between the captured instrument outputs and theplurality of musical symbols presented to the plurality of users; andpresenting, based on the level of correspondence, an adapted ornon-adapted sequence of music symbols at least one of the plurality ofplayers.

Example 90 includes the subject matter of example 89 and, optionally,wherein the at least one client device outputs accompanying backgroundmusic (BGM) that corresponds with the plurality of musical symbolspresented to the plurality of users.

Example 91 includes the subject matter of examples 89 and/or 90 and,optionally, wherein the cooperative playing of instruments by theplurality of users includes playing the instruments in timedcoordination (e.g., synchronously) or substantially synchronously, at asame location or at different locations.

Example 92 includes the subject matter of any one or more of theexamples 89 to 91 and, optionally, wherein the cooperative playing ofinstruments by the plurality of users includes playing the instrumentsin different time periods, at a same location or at different locations.

Example 93 includes the subject matter of Example 92 and, optionally,capturing the instrument outputs played by the plurality of users; andmerging the captured instrument outputs to arrive at a merged instrumentoutput.

Example 94 includes the subject matter of example 93 and, optionally,analyzing a level of correspondence of the merged instrument output withthe plurality of sequences of musical symbols presented to the users.

Example 95 includes the subject matter of any one or more of theexamples 89 to 94 and, optionally, wherein the instrument outputscomprise sounds captured by a microphone and/or audio signals capturedby a wired or wireless receiver device.

Example 96 pertains to a system for use with a musical piece, or partthereof, comprising: one or more processors; and one or more memoriesstoring software code portions executable by the one or more processorsto cause the system to perform the following: presenting, by at leastone client device to a plurality of users, a corresponding plurality ofsequences of musical symbols to be cooperatively played by the pluralityof users using a plurality of instruments; capturing instrument outputsproduced by the plurality of instruments to generate audio datadescriptive of the captured instrument outputs; analyzing the audio datafor determining a level of correspondence between the capturedinstrument outputs and the plurality of musical symbols presented to theplurality of users; and presenting, based on the level ofcorrespondence, an adapted or non-adapted sequence of music symbols atleast one of the plurality of players.

Example 97 includes the subject matter of example 96 and, optionally,wherein the at least one client device outputs accompanying backgroundmusic (BGM) that corresponds with the plurality of musical symbolspresented to the plurality of users.

Example 98 includes the subject matter of examples 96 and/or 97 and,optionally, wherein the cooperative playing of instruments by theplurality of users includes playing the instruments in timedcoordination (e.g., synchronously or substantially synchronously), at asame location or at different locations.

Example 99 includes the subject matter of any one or more of theExamples 96 to 97 and, optionally, wherein the cooperative playing ofinstruments by the plurality of users includes playing the instrumentsin different time periods, at a same location or at different locations.

Example 100 includes the subject matter of any one or more of examples96 to 99 and, optionally, capturing the instrument outputs played by theplurality of users (e.g., in a same time period or at different timeperiods, at a same location or different locations); and merging thecaptured instrument outputs to arrive at a merged instrument output.

Example 101 includes the subject matter of example 100 and, optionally,analyzing a level of correspondence of the merged instrument output withthe plurality of sequences of musical symbols presented to the users.

Example 102 includes the subject matter of any one or more of theexamples 96 to 101 and, optionally, wherein the instrument outputscomprise sounds captured by a microphone and/or audio signals capturedby a wired or wireless receiver device.

Example 103 pertains to a method for use with a musical piece, or partthereof, that is cooperatively played by a first musical instrumentaccording to first sequence of musical symbols, by a second musicalinstrument according to second sequence of musical symbols, and by athird musical instrument according to third sequence of musical symbols,for use with a first client device associated with the first musicalinstrument and operated by a first person that is identified by a firstperson identifier, with a second client device associated with thesecond musical instrument and operated by a second person that isidentified by a second person identifier, and with a third client deviceassociated with the third musical instrument and operated by a thirdperson that is identified by a third person identifier, wherein each ofthe first, second, and third client devices comprises one of thefollowing: a microphone, a digital signal connection, a sounder, and apresentation or display and communicates over the Internet and/or anyother communication network with a server device that stores the first,second, and third sequences, the method comprising: (e.g.,synchronously) sending, by the server device to the first, second, andthird client devices, respectively the first, second, and thirdsequences of musical symbols; receiving, by first, second, and thirdclient devices from the server device, respectively the first, second,and third sequences; displaying, to the first, second, and third personsby the respective display in the respective first, second, and thirdclient devices, the respective received first, second, and thirdsequences; capturing, by the respective microphone in the first, second,and third client devices, respective first, second, and third soundsfrom the respective first, second, and third musical instruments;sending, by the first, second, and third client devices to the serverdevice, the respective captured first, second, and third sounds;receiving, by the server device from the first, second, and third clientdevices, the respective captured first, second, and third sounds;analyzing, by the server device, the received captured first, second,and third sounds; sending, by the server device only to the second andthird client devices, the received captured first sound; receiving, bythe second and third client devices from the server device, the receivedcaptured first sound; emitting, by the sounder in the second and thirdclient devices, the received captured first sound; sending, by theserver device only to the third client device, the received capturedsecond sound; receiving, by the third client device from the serverdevice, the received captured second sound; and emitting, by the sounderin the third client devices, the received captured second sound.

Example 104 includes the subject matter of example 103 and, optionally,by the server device to the first client device, the received capturedsecond and third sounds; receiving, by the first client devices from theserver device, the received captured second and third sounds; andemitting, by the sounder in the first client device, the receivedcaptured second and third sounds.

Example 105 includes the subject matter of examples 103 and/or 104 and,optionally, sending, by the server device to the second client device,the received captured third sound; receiving, by the second clientdevice from the server device, the received captured third sound; andemitting, by the sounder in the second client device, the receivedcaptured third sound.

Example 106 includes the subject matter of any one or more of examples103 to 105 and, optionally, checking, by the server device, whether theeach of the received captured sounds matches the respective sequence;determining, by the server device, the amount of musical symbols in eachof the sequences that do not match with the respective captured sound;and updating, by the server device, the skill level value associatedwith the respective person identifier in response to the amount of themusical symbols that do not match in the respective captured sound

Example 107 includes the subject matter of example 106 and, optionally,for use with a threshold, and wherein the updating comprises raising theskill level value associated with the person identifier in response tothe amount of the non-matching musical symbols being less than thethreshold.

Example 108 includes the subject matter of examples 106 and, optionally,for use with a threshold, and wherein the updating comprises loweringthe skill level value associated with the person identifier in responseto the amount of the non-matching musical symbols being more than thethreshold.

The various features and steps discussed above, as well as other knownequivalents for each such feature or step, can be mixed and matched byone of ordinary skill in this art to perform methods in accordance withprinciples described herein. Although the disclosure has been providedin the context of certain embodiments and examples, it will beunderstood by those skilled in the art that the disclosure extendsbeyond the specifically described embodiments to other alternativeembodiments and/or uses and obvious modifications and equivalentsthereof. Accordingly, the disclosure is not intended to be limited bythe specific disclosures of embodiments herein.

Any digital computer system, module and/or engine exemplified herein canbe configured or otherwise programmed to implement a method disclosedherein, and to the extent that the system, module and/or engine isconfigured to implement such a method, it is within the scope and spiritof the disclosure. Once the system, module and/or engine are programmedto perform particular functions pursuant to computer readable andexecutable instructions from program software that implements a methoddisclosed herein, it in effect becomes a special purpose computerparticular to embodiments of the method disclosed herein. The methodsand/or processes disclosed herein may be implemented as a computerprogram product that may be tangibly embodied in an information carrierincluding, for example, in a non-transitory tangible computer-readableand/or non-transitory tangible machine-readable storage device. Thecomputer program product may directly loadable into an internal memoryof a digital computer, comprising software code portions for performingthe methods and/or processes as disclosed herein. The term“non-transitory” is used to exclude transitory, propagating signals, butto otherwise include any volatile or non-volatile computer memorytechnology suitable to the application.

Additionally or alternatively, the methods and/or processes disclosedherein may be implemented as a computer program that may be intangiblyembodied by a computer readable signal medium. A computer readablesignal medium may include a propagated data signal with computerreadable program code embodied therein, for example, in baseband or aspart of a carrier wave. Such a propagated signal may take any of avariety of forms, including, but not limited to, electro-magnetic,optical, or any suitable combination thereof. A computer readable signalmedium may be any computer readable medium that is not a non-transitorycomputer or machine-readable storage device and that can communicate,propagate, or transport a program for use by or in connection withapparatuses, systems, platforms, methods, operations and/or processesdiscussed herein.

The terms “non-transitory computer-readable storage device” and“non-transitory machine-readable storage device” encompassesdistribution media, intermediate storage media, execution memory of acomputer, and any other medium or device capable of storing for laterreading by a computer program implementing embodiments of a methoddisclosed herein. A computer program product can be deployed to beexecuted on one computer or on multiple computers at one site ordistributed across multiple sites and interconnected by one or morecommunication networks.

These computer readable and executable instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable and executable programinstructions may also be stored in a computer readable storage mediumthat can direct a computer, a programmable data processing apparatus,and/or other devices to function in a particular manner, such that thecomputer readable storage medium having instructions stored thereincomprises an article of manufacture including instructions whichimplement aspects of the function/act specified in the flowchart and/orblock diagram block or blocks.

The computer readable and executable instructions may also be loadedonto a computer, other programmable data processing apparatus, or otherdevice to cause a series of operational steps to be performed on thecomputer, other programmable apparatus or other device to produce acomputer implemented process, such that the instructions which executeon the computer, other programmable apparatus, or other device implementthe functions/acts specified in the flowchart and/or block diagram blockor blocks.

In the discussion, unless otherwise stated, adjectives such as“substantially” and “about” that modify a condition or relationshipcharacteristic of a feature or features of an embodiment of theinvention, are to be understood to mean that the condition orcharacteristic is defined to within tolerances that are acceptable foroperation of the embodiment for an application for which it is intended.

Unless otherwise specified, the terms ‘about’ and/or ‘close’ withrespect to a magnitude or a numerical value may imply to be within aninclusive range of −10% to +10% of the respective magnitude or value.

It should be noted that where an embodiment refers to a condition of“above a threshold”, this should not be construed as excluding anembodiment referring to a condition of “equal or above a threshold”.Analogously, where an embodiment refers to a condition “below athreshold”, this should not to be construed as excluding an embodimentreferring to a condition “equal or below a threshold”. It is clear thatshould a condition be interpreted as being fulfilled if the value of agiven parameter is above a threshold, then the same condition isconsidered as not being fulfilled if the value of the given parameter isequal or below the given threshold. Conversely, should a condition beinterpreted as being fulfilled if the value of a given parameter isequal or above a threshold, then the same condition is considered as notbeing fulfilled if the value of the given parameter is below (and onlybelow) the given threshold.

It should be understood that where the claims or specification refer to“a” or “an” element and/or feature, such reference is not to beconstrued as there being only one of that element. Hence, reference to“an element” or “at least one element” for instance may also encompass“one or more elements”.

As used herein the term “configuring” and/or ‘adapting’ for anobjective, or a variation thereof, implies using materials and/orcomponents in a manner designed for and/or implemented and/or operableor operative to achieve the objective.

Unless otherwise stated or applicable, the use of the expression“and/or” between the last two members of a list of options for selectionindicates that a selection of one or more of the listed options isappropriate and may be made, and may be used interchangeably with theexpressions “at least one of the following”, “any one of the following”or “one or more of the following”, followed by a listing of the variousoptions.

As used herein, the phrase “A,B,C, or any combination of the aforesaid”should be interpreted as meaning all of the following: (i) A or B or Cor any combination of A, B, and C, (ii) at least one of A, B, and C; and(iii) A, and/or B and/or C. This concept is illustrated for threeelements (i.e., A,B,C), but extends to fewer and greater numbers ofelements (e.g., A, B, C, D, etc.).

It is noted that the terms “operable to” or “operative to” can encompassthe meaning of the term “adapted or configured to”. In other words, amachine “operable to” or “operative to” perform a task can in someembodiments, embrace a mere capability (e.g., “adapted”) to perform thefunction and, in some other embodiments, a machine that is actually made(e.g., “configured”) to perform the function.

Throughout this application, various embodiments of this invention maybe presented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of theinvention. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible subranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed subranges such as from 1 to 4, from 2 to 4, from 1 to 5, from2 to 5, from 2 to 6, from 4 to 6 etc., as well as individual numberswithin that range, for example, 1, 2, 3, 4, 5, and 6. This appliesregardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to includeany cited numeral (fractional or integral) within the indicated range.The phrases “ranging/ranges between” a first indicate number and asecond indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all thefractional and integral numerals therebetween.

It should be appreciated that combination of features disclosed indifferent embodiments are also included within the scope of the presentinventions.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents will now occur to those of ordinary skill in the art. It is,therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the invention.

Any method herein may be in combination with a Virtual Reality (VR)system that may simulate a virtual environment to the person, and anycommunication with the VR system may be wireless. Any VR system hereinmay comprise a Head-Mounted Display (HMD), and any client device hereinmay comprise, may be part of, may consist of, or may be integrated with,the HMD.

Any device herein may serve as a client device in the meaning ofclient/server architecture, commonly initiating requests for receivingservices, functionalities, and resources, from other devices (servers orclients). Each of the these devices may further employ, store,integrate, or operate a client-oriented (or end-point dedicated)operating system, such as Microsoft Windows® (including the variants:Windows 7, Windows XP, Windows 8, and Windows 8.1, available fromMicrosoft Corporation, headquartered in Redmond, Wash., U.S.A.), Linux,and Google Chrome OS available from Google Inc. headquartered inMountain View, Calif., U.S.A. Further, each of the these devices mayfurther employ, store, integrate, or operate a mobile operating systemsuch as Android (available from Google Inc. and includes variants suchas version 2.2 (Froyo), version 2.3 (Gingerbread), version 4.0 (IceCream Sandwich), Version 4.2 (Jelly Bean), and version 4.4 (KitKat),Android version 6.0 (Marshmallow), Android version 7.0 (Nougat), Androidversion 8.0 (Oreo), Android version 9.0 (Pie), Android 10, Android 11,iOS (available from Apple Inc., and includes variants such as versions3-7), Apple iOS version 8, Apple iOS version 9, Apple iOS version 10,Apple iOS version 11, Apple iOS version 12, Apple iOS version 13, AppleiOS version 14, Windows® Phone (available from Microsoft Corporation andincludes variants such as version 7, version 8, or version 9), orBlackberry® operating system (available from BlackBerry Ltd.,headquartered in Waterloo, Ontario, Canada). Alternatively or inaddition, each of the devices that are not denoted herein as servers mayequally function as a server in the meaning of client/serverarchitecture. Any one of the servers herein may be a web server usingHyper Text Transfer Protocol (HTTP) that responds to HTTP requests viathe Internet, and any request herein may be an HTTP request. AnyOperating System (OS) herein, such as any server or client operatingsystem, may consists of, include, or be based on a real-time operatingsystem (RTOS), such as FreeRTOS, SafeRTOS, QNX, VxWorks, orMicro-Controller Operating Systems (μC/OS).

Examples of web browsers include Microsoft Internet Explorer (availablefrom Microsoft Corporation, headquartered in Redmond, Wash., U.S.A.),Google Chrome that is a freeware web browser (developed by Google,headquartered in Googleplex, Mountain View, Calif., U.S.A.), Opera™(developed by Opera Software ASA, headquartered in Oslo, Norway), andMozilla Firefox® (developed by Mozilla Corporation headquartered inMountain View, Calif., U.S.A.). The web-browser may be a mobile browser,such as Safari (developed by Apple Inc. headquartered in Apple Campus,Cupertino, Calif., U.S.A.), Opera Mini™ (developed by Opera SoftwareASA, headquartered in Oslo, Norway), and Android web browser.

Any of the steps described herein for any method herein may besequential, and performed in the described order. For example, in a casewhere a step is performed in response to another step, or uponcompletion of another step, the steps are executed one after the other.However, in the case where two or more steps are not explicitlydescribed as being sequentially executed, these steps may be executed inany order, or may be simultaneously performed. Two or more steps may beexecuted by two different network elements, or in the same networkelement, and may be executed in parallel using multiprocessing ormultitasking.

Any networking protocol may be utilized for exchanging informationbetween the network elements (e.g., clients or servers) within thenetwork (such as the Internet). For example, it is contemplated thatcommunications can be performed using TCP/IP. Generally, HTTP and HTTPSare utilized on top of TCP/IP as the message transport envelope. Thesystem described hereinafter is suited for both HTTP/HTTPS,message-queuing systems, and other communications transport protocoltechnologies. Furthermore, depending on the differing business andtechnical requirements of the various partners within the network, thephysical network may embrace and utilize multiple communication protocoltechnologies. As used herein, the term “request” includes, but is notlimited to, a message describing an operation to be carried out in thecontext of a specified resource, such as HTTP GET, POST, PUT, and HEADcommands, and the term “response” includes, but is not limited to, amessage containing the result of an executed request, such as an HTMLdocument or a server error message. A request may be an explicit webrequest that is initiated manually by the user, or may be an implicitrequest that is initiated by a web client and is transparent to theuser, as an ancillary event corresponding to an explicit web request.

Application software is typically a set of one, or more programsdesigned to carry out operations for a specific application. Commonly,an application software is dependent on system software that manages andintegrates computer capabilities, but does not directly perform tasksthat benefit the user, such as an operating system, to execute. Examplesof types of application software may include accounting software, mediaplayers, and office suites. Applications may be bundled with thecomputer and its system software, or may be published separately, andfurther may be developed and coded as a proprietary, or as anopen-source software. Most applications are designed to help peopleperform an activity.

Where certain process steps are described in a particular order or wherealphabetic and/or alphanumeric labels are used to identify certainsteps, the embodiments are not limited to any particular order ofcarrying out such steps. In particular, the labels are used merely forconvenient identification of steps, and are not intended to imply,specify or require a particular order for carrying out such steps.Furthermore, other embodiments may use more or less steps than thosediscussed herein. They may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network.

In a distributed computing environment, program modules may be locatedin both local and remote memory storage devices. Any single step, groupof steps, or a flow chart herein may be realized as a computer programin a centralized fashion, in one computer system or in a distributedfashion where different elements are spread across severalinterconnected computer systems. Any kind of computer system, or otherapparatus adapted for carrying out the methods described herein, issuited to perform the functions described herein. A typical centralizedimplementation could include a general purpose computer system with acomputer program that, when being loaded and executed, will control thecomputer system, and carry out the methods described herein.

Computer program or application in the present context means anyexpression, in any language, code or notation, of a set of instructionsintended to cause a system having an information processing capabilityto perform a particular function either directly or after either, orboth of the following a) conversion to another language, code ornotation; b) reproduction in a different material form. Certain aspectsof the described techniques include process steps and instructionsdescribed herein in the form of an algorithm. It should be noted thatthe described process steps and instructions could be embodied insoftware, firmware or hardware, and when embodied in software, could bedownloaded to reside on and be operated from different platforms used byreal time network operating systems. Some portions of the abovedescription present the techniques described herein in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are used by thoseskilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. These operations,while described functionally or logically, are understood to beimplemented by computer programs. Furthermore, it has also provenconvenient at times to refer to these arrangements of operations asmodules or by functional names, without loss of generality.

The corresponding structures, materials, acts, and equivalents of allmeans plus function elements in the claims below are intended to includeany structure, or material, for performing the function in combinationwith other specifically claimed elements. The description of the presentinvention has been presented for purposes of illustration anddescription, but is not intended to be exhaustive or limited to theinvention in the form disclosed. The present invention should not beconsidered limited to the particular embodiments described above, butrather should be understood to cover all aspects of the invention asfairly set out in the attached claims. Various modifications, equivalentprocesses, as well as numerous structures to which the present inventionmay be applicable, will be readily apparent to those skilled in the artto which the present invention is directed upon review of the presentdisclosure.

Any apparatus herein, which may be any of the systems, devices, modules,or functionalities described herein, may be integrated with asmartphone. The integration may be by being enclosed in the samehousing, sharing a power source (such as a battery), using the sameprocessor, or any other integration functionality. In one example, thefunctionality of any apparatus herein, which may be any of the systems,devices, modules, or functionalities described here, is used to improve,to control, or otherwise be used by the smartphone. In one example, ameasured or calculated value by any of the systems, devices, modules, orfunctionalities described herein, is output to the smartphone device orfunctionality to be used therein. Alternatively or in addition, any ofthe systems, devices, modules, or functionalities described herein isused as a sensor for the smartphone device or functionality.

A ‘nominal’ value herein refers to a designed, expected, or targetvalue. In practice, a real or actual value is used, obtained, or exists,which varies within a tolerance from the nominal value, typicallywithout significantly affecting functioning. Common tolerances are 20%,15%, 10%, 5%, or 1% around the nominal value.

Discussions herein utilizing terms such as, for example, “processing,”“computing”, “calculating”, “determining”, “establishing”, “analyzing”,“checking”, or the like, may refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulate and/or transform datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information storage medium that may storeinstructions to perform operations and/or processes.

Some portions are presented in terms of algorithms or symbolicrepresentations of operations on data bits or binary digital signalsstored within a computing system memory, such as a computer memory.These algorithmic descriptions or representations are examples oftechniques used by those of ordinary skill in the data processing art toconvey the substance of their work to others skilled in the art. Analgorithm is a self-consistent sequence of operations or similarprocessing leading to a desired result. In this context, operations orprocessing involves physical manipulation of physical quantities.Typically, although not necessarily, such quantities may take the formof electrical or magnetic signals capable of being stored, transferred,combined, compared, or otherwise manipulated. It has proven convenientat times, principally for reasons of common usage, to refer to suchsignals as bits, data, values, elements, symbols, characters, terms,numbers, numerals, or the like. It should be understood, however, thatall of these and similar terms are to be associated with appropriatephysical quantities and are merely convenient labels. Unlessspecifically stated otherwise, it is appreciated that throughout thisspecification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical, electronic, ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

Throughout the description and claims of this specification, the word“couple” and variations of that word such as “coupling”, “coupled”, and“couplable”, refers to an electrical connection (such as a copper wireor soldered connection), a logical connection (such as through logicaldevices of a semiconductor device), a virtual connection (such asthrough randomly assigned memory locations of a memory device) or anyother suitable direct or indirect connections (including combination orseries of connections), for example, for allowing the transfer of power,signal, or data, as well as connections formed through interveningdevices or elements.

The arrangements and methods described herein may be implemented usinghardware, software or a combination of both. The term “integration” or“software integration” or any other reference to the integration of twoprograms or processes herein refers to software components (e.g.,programs, modules, functions, processes etc.) that are (directly or viaanother component) combined, working or functioning together or form awhole, commonly for sharing a common purpose or a set of objectives.Such software integration can take the form of sharing the same programcode, exchanging data, being managed by the same manager program,executed by the same processor, stored on the same medium, sharing thesame GUI or other user interface, sharing peripheral hardware (such as amonitor, printer, keyboard and memory), sharing data or a database, orbeing part of a single package. The term “integration” or “hardwareintegration” or integration of hardware components herein refers tohardware components that are (directly or via another component)combined, working or functioning together or form a whole, commonly forsharing a common purpose or set of objectives. Such hardware integrationcan take the form of sharing the same power source (or power supply) orsharing other resources, exchanging data or control (e.g., bycommunicating), being managed by the same manager, physically connectedor attached, sharing peripheral hardware connection (such as a monitor,printer, keyboard and memory), being part of a single package or mountedin a single enclosure (or any other physical collocating), sharing acommunication port, or used or controlled with the same software orhardware. The term “integration” herein refers (as applicable) to asoftware integration, a hardware integration, or any combinationthereof.

As used herein, the term “portable” herein refers to physicallyconfigured to be easily carried or moved by a person of ordinarystrength using one or two hands, without the need for any specialcarriers.

Any mechanical attachment of joining two parts herein refers toattaching the parts with sufficient rigidity to prevent unwantedmovement between the attached parts. Any type of fastening means may beused for the attachments, including chemical material such as anadhesive or a glue, or mechanical means such as screw or bolt. Anadhesive (used interchangeably with glue, cement, mucilage, or paste) isany substance applied to one surface, or both surfaces, of two separateitems that binds them together and resists their separation. Adhesivematerials may be reactive and non-reactive adhesives, which refers towhether the adhesive chemically reacts in order to harden, and their rawstock may be of natural or synthetic origin.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provides a resultconditioned on one or more inputs. Suitable computing devices includemultipurpose microprocessor-based computer systems accessing storedsoftware that programs or configures the computing system from ageneral-purpose computing apparatus to a specialized computing apparatusimplementing one or more embodiments of the present subject matter. Anysuitable programming, scripting, or other type of language orcombinations of languages may be used to implement the teachingscontained herein in software to be used in programming or configuring acomputing device.

As used herein, the term “Integrated Circuit” (IC) shall include anytype of integrated device of any function where the electronic circuitis manufactured by the patterned diffusion of trace elements into thesurface of a thin substrate of semiconductor material (e.g., Silicon),whether single or multiple die, or small or large scale of integration,and irrespective of process or base materials (including, withoutlimitation Si, SiGe, CMOS and GAs) including, without limitation,applications specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital processors (e.g., DSPs, CISCmicroprocessors, or RISC processors), so-called “system-on-a-chip” (SoC)devices, memory (e.g., DRAM, SRAM, flash memory, ROM), mixed-signaldevices, and analog ICs.

The circuits in an IC are typically contained in a silicon piece or in asemiconductor wafer, and commonly packaged as a unit. The solid-statecircuits commonly include interconnected active and passive devices,diffused into a single silicon chip. Integrated circuits can beclassified into analog, digital and mixed signal (both analog anddigital on the same chip). Digital integrated circuits commonly containmany of logic gates, flip-flops, multiplexers, and other circuits in afew square millimeters. The small size of these circuits allows highspeed, low power dissipation, and reduced manufacturing cost comparedwith board-level integration. Further, a multi-chip module (MCM) may beused, where multiple integrated circuits (ICs), the semiconductor dies,or other discrete components are packaged onto a unifying substrate,facilitating their use as a single component (as though a larger IC).

The term “computer-readable medium” (or “machine-readable medium”) asused herein is an extensible term that refers to any non-transitorycomputer readable medium or any memory, that participates in providinginstructions to a processor (such as processor 12) for execution, or anymechanism for storing or transmitting information in a form readable bya machine (e.g., a computer). Such a medium may storecomputer-executable instructions to be executed by a processing elementand/or software, and data that is manipulated by a processing elementand/or software, and may take many forms, including but not limited to,non-volatile medium, volatile medium, and transmission medium.Transmission media includes coaxial cables, copper wire and fiberoptics. Transmission media can also take the form of acoustic or lightwaves, such as those generated during radio-wave and infrared datacommunications, or other form of propagating signals (e.g., carrierwaves, infrared signals, digital signals, etc.). Common forms ofcomputer-readable media include, for example, a floppy disk, a flexibledisk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM,any other optical medium, punch-cards, paper-tape, any other physicalmedium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave as describedhereinafter, or any other medium from which a computer can read. Anycomputer-readable storage medium herein, such as the main memory 15 a,the storage device 15 c, the ROM 15 b, or the storage 33, may include asingle medium or multiple media (e.g., a centralized or distributeddatabase and/or associated caches and servers) that store the one ormore sets of instructions. The term “computer-readable storage medium”may also include any medium that is capable of storing, encoding orcarrying a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methods of thepresent disclosure. The term “computer-readable storage medium” mayaccordingly be taken to include, but not be limited to, solid-statememories, optical media and magnetic media.

Computer-executable instructions may include, for example, instructionsand data, which cause a general purpose computer, special purposecomputer, or special purpose processing device (e.g., one or moreprocessors) to perform a certain function or group of functions.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

As used herein, the terms “module” or “component” may refer to specifichardware implementations configured to perform the operations of themodule or component and/or software objects or software routines thatmay be stored on and/or executed by general purpose hardware (e.g.,computer-readable media, processing devices, etc.) of the computingsystem. In some embodiments, the different components, modules, engines,and services described herein may be implemented as objects or processesthat execute on the computing system (e.g., as separate threads). Whilesome of the system and methods described herein are generally describedas being implemented in software (stored on and/or executed by generalpurpose hardware), specific hardware implementations or a combination ofsoftware and specific hardware implementations are also possible andcontemplated. In this description, a “computing entity” may be anycomputing system as previously defined herein, or any module orcombination of modulates running on a computing system.

Any process descriptions or blocks in any logic flowchart herein shouldbe understood as representing modules, segments, portions of code, orsteps that include one or more instructions for implementing specificlogical functions in the process, and alternative implementations areincluded within the scope of the present invention in which functionsmay be executed out of order from that shown or discussed, includingsubstantially concurrently or in reverse order, depending on thefunctionality involved, as would be understood by those reasonablyskilled in the art of the present invention.

Each of the methods or steps herein, may consist of, include, be partof, be integrated with, or be based on, a part of, or the whole of, thesteps, functionalities, or structure (such as software) described in thepublications that are incorporated in their entirety herein. Further,each of the components, devices, or elements herein may consist of,integrated with, include, be part of, or be based on, a part of, or thewhole of, the components, systems, devices or elements described in thepublications that are incorporated in their entirety herein.

Any part of, or the whole of, any of the methods described herein may beprovided as part of, or used as, an Application Programming Interface(API), defined as an intermediary software serving as the interfaceallowing the interaction and data sharing between an applicationsoftware and the application platform, across which few or all servicesare provided, and commonly used to expose or use a specific softwarefunctionality, while protecting the rest of the application. The API maybe based on, or according to, Portable Operating System Interface(POSIX) standard, defining the API along with command line shells andutility interfaces for software compatibility with variants of Unix andother operating systems, such as POSIX.1-2008 that is simultaneouslyIEEE STD. 1003.1™ —2008 entitled: “Standard for InformationTechnology—Portable Operating System Interface (POSIX® Description”, andThe Open Group Technical Standard Base Specifications, Issue 7, IEEESTD. 1003.1™, 2013 Edition.

The term “computer” is used generically herein to describe any number ofcomputers, including, but not limited to personal computers, embeddedprocessing elements and systems, software, ASICs, chips, workstations,mainframes, etc. Any computer herein may consist of, or be part of, ahandheld computer, including any portable computer that is small enoughto be held and operated while holding in one hand or fit into a pocket.Such a device, also referred to as a mobile device, typically has adisplay screen with touch input and/or miniature keyboard. Non-limitingexamples of such devices include a Digital Still Camera (DSC), a Digitalvideo Camera (DVC or digital camcorder), a Personal Digital Assistant(PDA), and mobile phones and Smartphones. The mobile devices may combinevideo, audio and advanced communication capabilities, such as PAN andWLAN. A mobile phone (also known as a cellular phone, cell phone and ahand phone) is a device that can make and receive telephone calls over aradio link whilst moving around a wide geographic area, by connecting toa cellular network provided by a mobile network operator. The calls areto and from the public telephone network, which includes other mobilesand fixed-line phones across the world. The Smartphones may combine thefunctions of a Personal Digital Assistant (PDA), and may serve asportable media players and camera phones with high-resolutiontouch-screens, web browsers that can access, and properly display,standard web pages rather than just mobile-optimized sites, GPSnavigation, Wi-Fi and mobile broadband access. In addition to telephony,the Smartphones may support a wide variety of other services such astext messaging, MMS, email, Internet access, short-range wirelesscommunications (infrared, Bluetooth), business applications, gaming andphotography.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations can be expressly set forth herein for sakeof clarity.

Any selection herein, such as any selection from any list or group, maybe based on, or may use, a load balancing, a First-In-First-Out (FIFO)scheme, a Last-In-First-Out (LIFO) scheme, a sequential or cyclicselection, a random selection, or any combination thereof. Any randomselection herein may use, or may be based on, one or more random numbersgenerated by a random number generator. Any random number generatorherein may be hardware based, and may be using thermal noise, shotnoise, nuclear decaying radiation, photoelectric effect, or quantumphenomena. Alternatively or in addition, any random number generatorherein may be software based, and may be based on executing an algorithmfor generating pseudo-random numbers.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims can contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations. In addition, even if a specificnumber of an introduced claim recitation is explicitly recited, thoseskilled in the art will recognize that such recitation should beinterpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, means at leasttwo recitations, or two or more recitations). Furthermore, in thoseinstances where a convention analogous to “at least one of A, B, and C,etc.” is used, in general such a construction is intended in the senseone having skill in the art would understand the convention (e.g., “asystem having at least one of A, B, and C” would include but not belimited to systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.). In those instances where a convention analogous to “atleast one of A, B, or C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (e.g., “a system having at least one of A, B, or C” wouldinclude but not be limited to systems that have A alone, B alone, Calone, A and B together, A and C together, B and C together, and/or A,B, and C together, etc.). It will be further understood by those withinthe art that virtually any disjunctive word and/or phrase presenting twoor more alternative terms, whether in the description, claims, ordrawings, should be understood to contemplate the possibilities ofincluding one of the terms, either of the terms, or both terms. Forexample, the phrase “A or B” will be understood to include thepossibilities of “A” or “B” or “A and B.”

As will be understood by one skilled in the art, for any and allpurposes, such as in terms of providing a written description, allranges disclosed herein also encompass any and all possible subrangesand combinations of subranges thereof. Any listed range can be easilyrecognized as sufficiently describing and enabling the same range beingbroken down into at least equal halves, thirds, quarters, fifths,tenths, etc. As a non-limiting example, each range discussed herein canbe readily broken down into a lower third, middle third and upper third,etc. As will also be understood by one skilled in the art all languagesuch as “up to,” “at least,” and the like include the number recited andrefer to ranges which can be subsequently broken down into subranges asdiscussed above. Finally, as will be understood by one skilled in theart, a range includes each individual member. Thus, for example, a grouphaving 1-3 cells refers to groups having 1, 2, or 3 cells. Similarly, agroup having 1-5 cells refers to groups having 1, 2, 3, 4, or 5 cells,and so forth.

Some embodiments may be used in conjunction with various devices andsystems, for example, a Personal Computer (PC), a desktop computer, amobile computer, a laptop computer, a notebook computer, a tabletcomputer, a server computer, a handheld computer, a handheld device, aPersonal Digital Assistant (PDA) device, a cellular handset, asmartphone, a tablet, a handheld PDA device, an on-board device, anoff-board device, a hybrid device, a vehicular device, a non-vehiculardevice, a mobile or portable device, a non-mobile or non-portabledevice, a wireless communication station, a wireless communicationdevice, a wireless Access Point (AP), a wired or wireless router, awired or wireless modem, a wired or wireless network, a Local AreaNetwork (LAN), a Wireless LAN (WLAN), a Metropolitan Area Network (MAN),a Wireless MAN (WMAN), a Wide Area Network (WAN), a Wireless WAN (WWAN),a Personal Area Network (PAN), a Wireless PAN (WPAN), devices and/ornetworks operating substantially in accordance with existing IEEE802.11, 802.11a, 802.11b, 802.11g, 802.11k, 802.11n, 802.11r, 802.16,802.16d, 802.16e, 802.20, 802.21 standards and/or future versions and/orderivatives of the above standards, units and/or devices that are partof the above networks, one way and/or two-way radio communicationsystems, cellular radio-telephone communication systems, a cellulartelephone, a wireless telephone, a Personal Communication Systems (PCS)device, a PDA device that incorporates a wireless communication device,a mobile or portable GNSS such as the Global Positioning System (GPS)device, a device that incorporates a GNSS or GPS receiver or transceiveror chip, a device that incorporates an RFID element or chip, a MultipleInput Multiple Output (MIMO) transceiver or device, a Single InputMultiple Output (SIMO) transceiver or device, a Multiple Input SingleOutput (MISO) transceiver or device, a device having one or moreinternal antennas and/or external antennas, Digital Video Broadcast(DVB) devices or systems, multi-standard radio devices or systems, awired or wireless handheld device (e.g., BlackBerry, Palm Treo), aWireless Application Protocol (WAP) device, or the like.

Any system or apparatus herein may further be operative for storing,operating, or using, an operating system. Any system herein may comprisea Virtual Machine (VM) for virtualization, and the operating system maybe executed as a guest operating system. Any system herein may furthercomprise a host computer that implements the VM, and the host computermay be operative for executing a hypervisor or a Virtual Machine Monitor(VMM), and the guest operating system may use or may interface virtualhardware. Any virtualization herein, such as any operating systemvirtualization, may include, may be based on, or may use, fullvirtualization, para-virtualization, or hardware assistedvirtualization.

As used herein, the terms “program”, “programmable”, and “computerprogram” are meant to include any sequence or human or machinecognizable steps, which perform a function. Such programs are notinherently related to any particular computer or other apparatus, andmay be rendered in virtually any programming language or environment,including, for example, C/C++, Fortran, COBOL, PASCAL, Assemblylanguage, markup languages (e.g., HTML, SGML, XML, VoXML), and the like,as well as object-oriented environments, such as the Common ObjectRequest Broker Architecture (CORBA), Java™ (including J2ME, Java Beans,etc.) and the like, as well as in firmware or other implementations.Generally, program modules include routines, subroutines, procedures,definitional statements and macros, programs, objects, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. A compiler may be used to create an executablecode, or a code may be written using interpreted languages such as PERL,Python, or Ruby.

The terms “task” and “process” are used generically herein to describeany type of running programs, including, but not limited to a computerprocess, task, thread, executing application, operating system, userprocess, device driver, native code, machine or other language, etc.,and can be interactive and/or non-interactive, executing locally and/orremotely, executing in foreground and/or background, executing in theuser and/or operating system address spaces, a routine of a libraryand/or standalone application, and is not limited to any particularmemory partitioning technique. The steps, connections, and processing ofsignals and information illustrated in the figures, including, but notlimited to, any block and flow diagrams and message sequence charts, maytypically be performed in the same or in a different serial or parallelordering and/or by different components and/or processes, threads, etc.,and/or over different connections and be combined with other functionsin other embodiments, unless this disables the embodiment or a sequenceis explicitly or implicitly required (e.g., for a sequence of readingthe value, processing the value: the value must be obtained prior toprocessing it, although some of the associated processing may beperformed prior to, concurrently with, and/or after the read operation).Where certain process steps are described in a particular order or wherealphabetic and/or alphanumeric labels are used to identify certainsteps, the embodiments of the invention are not limited to anyparticular order of carrying out such steps. In particular, the labelsare used merely for convenient identification of steps, and are notintended to imply, specify or require a particular order for carryingout such steps. Furthermore, other embodiments may use more or lesssteps than those discussed herein. The invention may also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

As used in this application, the term “about” or “approximately” refersto a range of values within plus or minus 10% of the specified number.As used in this application, the term “substantially” means that theactual value is within about 10% of the actual desired value,particularly within about 5% of the actual desired value and especiallywithin about 1% of the actual desired value of any variable, element orlimit set forth herein.

The corresponding structures, materials, acts, and equivalents of allmeans plus function elements in the claims below are intended to includeany structure, or material, for performing the function in combinationwith other claimed elements as specifically claimed. The description ofthe present invention has been presented for purposes of illustrationand description, but is not intended to be exhaustive or limited to theinvention in the form disclosed. The present invention should not beconsidered limited to the particular embodiments described above, butrather should be understood to cover all aspects of the invention asfairly set out in the attached claims. Various modifications, equivalentprocesses, as well as numerous structures to which the present inventionmay be applicable, will be readily apparent to those skilled in the artto which the present invention is directed upon review of the presentdisclosure.

All publications, standards, patents, and patent applications cited inthis specification are incorporated herein by reference as if eachindividual publication, patent, or patent application were specificallyand individually indicated to be incorporated by reference and set forthin its entirety herein.

What is claimed is:
 1. A method for use with a musical piece, or partthereof, comprising: presenting, by at least one client device to aplurality of users, a corresponding plurality of sequences of musicalsymbols to be cooperatively played by the plurality of users using aplurality of instruments; capturing instrument outputs produced by theplurality of instruments to generate audio data descriptive of thecaptured instrument outputs; analyzing the audio data for determining alevel of correspondence between the captured instrument outputs and theplurality of musical symbols presented to the plurality of users; andpresenting, based on the level of correspondence, an adapted ornon-adapted sequence of music symbols to at least one of the pluralityof players.
 2. The method of claim 1, wherein the at least one clientdevice outputs accompanying background music (BGM) that corresponds withthe plurality of musical symbols presented to the plurality of users. 3.The method of claim 1, wherein the cooperative playing of instruments bythe plurality of users includes playing the instruments in timedcoordination, at a same location or at different locations.
 4. Themethod of claim 1, wherein the cooperative playing of instruments by theplurality of users includes playing the instruments in different timeperiods, at a same location or at different locations.
 5. The method ofclaim 1, further comprising: capturing the instrument outputs played bythe plurality of users; merging the captured instrument outputs toarrive at a merged instrument output.
 6. The method of claim 5, furthercomprising analyzing a level of correspondence of the merged instrumentoutput with the plurality of sequences of musical symbols presented tothe users.
 7. The method of claim 1, wherein the instrument outputscomprise sounds captured by a microphone and/or audio signals capturedby a wired or wireless receiver device.
 8. A system for use with amusical piece, or part thereof, comprising: one or more processors; andone or more memories storing software code portions executable by theone or more processors to cause the system to perform the following:presenting, by at least one client device to a plurality of users, acorresponding plurality of sequences of musical symbols to becooperatively played by the plurality of users using a plurality ofinstruments; capturing instrument outputs produced by the plurality ofinstruments to generate audio data descriptive of the capturedinstrument outputs; analyzing the audio data for determining a level ofcorrespondence between the captured instrument outputs and the pluralityof musical symbols presented to the plurality of users; and presenting,based on the level of correspondence, an adapted or non-adapted sequenceof music symbols at least one of the plurality of players
 9. The systemof claim 8, wherein the at least one client device outputs accompanyingbackground music (BGM) that corresponds with the plurality of musicalsymbols presented to the plurality of users.
 10. The system of claim 8,wherein the cooperative playing of instruments by the plurality of usersincludes playing the instruments in timed coordination, at a samelocation or at different locations.
 11. The system of claim 8, whereinthe cooperative playing of instruments by the plurality of usersincludes playing the instruments in different time periods, at a samelocation or at different locations.
 12. The system of claim 8, furthercomprising: capturing the instrument outputs played by the plurality ofusers; merging the captured instrument outputs to arrive at a mergedinstrument output.
 13. The system of claim 12, further comprisinganalyzing a level of correspondence of the merged instrument output withthe plurality of sequences of musical symbols presented to the users.14. The system of claim 8, wherein the instrument outputs comprisesounds captured by a microphone and/or audio signals captured by a wiredor wireless receiver device
 15. A method for use with a musical piece,or part thereof, that is cooperatively played by a first musicalinstrument according to first sequence of musical symbols, by a secondmusical instrument according to second sequence of musical symbols, andby a third musical instrument according to third sequence of musicalsymbols, for use with a first client device associated with the firstmusical instrument and operated by a first person that is identified bya first person identifier, with a second client device associated withthe second musical instrument and operated by a second person that isidentified by a second person identifier, and with a third client deviceassociated with the third musical instrument and operated by a thirdperson that is identified by a third person identifier, wherein each ofthe first, second, and third client devices comprises one of thefollowing: a microphone, a digital signal connection, a sounder, and apresentation or display and communicates over the Internet and/or anyother communication network with a server device that stores the first,second, and third sequences, the method comprising: sending, by theserver device to the first, second, and third client devices,respectively the first, second, and third sequences of musical symbols;receiving, by first, second, and third client devices from the serverdevice, respectively the first, second, and third sequences; displaying,to the first, second, and third persons by the respective display in therespective first, second, and third client devices, the respectivereceived first, second, and third sequences; capturing, by therespective microphone in the first, second, and third client devices,respective first, second, and third sounds from the respective first,second, and third musical instruments; sending, by the first, second,and third client devices to the server device, the respective capturedfirst, second, and third sounds; receiving, by the server device fromthe first, second, and third client devices, the respective capturedfirst, second, and third sounds; analyzing, by the server device, thereceived captured first, second, and third sounds; sending, by theserver device only to the second and third client devices, the receivedcaptured first sound; receiving, by the second and third client devicesfrom the server device, the received captured first sound; emitting, bythe sounder in the second and third client devices, the receivedcaptured first sound; sending, by the server device only to the thirdclient device, the received captured second sound; receiving, by thethird client device from the server device, the received captured secondsound; and emitting, by the sounder in the third client devices, thereceived captured second sound.
 16. The method according to claim 15,further comprising: sending, by the server device to the first clientdevice, the received captured second and third sounds; receiving, by thefirst client devices from the server device, the received capturedsecond and third sounds; and emitting, by the sounder in the firstclient device, the received captured second and third sounds.
 17. Themethod according to claim 15, further comprising: sending, by the serverdevice to the second client device, the received captured third sound;receiving, by the second client device from the server device, thereceived captured third sound; and emitting, by the sounder in thesecond client device, the received captured third sound.
 18. The methodaccording to claim 15, further comprising: checking, by the serverdevice, whether the each of the received captured sounds matches therespective sequence; determining, by the server device, the amount ofmusical symbols in each of the sequences that do not match with therespective captured sound; and updating, by the server device, the skilllevel value associated with the respective person identifier in responseto the amount of the musical symbols that do not match in the respectivecaptured sound.
 19. The method according to claim 18, for use with athreshold, and wherein the updating comprises raising the skill levelvalue associated with the person identifier in response to the amount ofthe non-matching musical symbols being less than the threshold.
 20. Themethod according to claim 18, for use with a threshold, and wherein theupdating comprises lowering the skill level value associated with theperson identifier in response to the amount of the non-matching musicalsymbols being more than the threshold.